In contrast with the conventional DHN model (Hopfield, 1982), where neural states are defined as binary variables, in a CHN they are continuous (Hopfield, 1984). The dynamics of the network is defined by a set of differential equations with each neuron unit described as a leaky integration:

where u_i is the membrane potential of neuron i, c is the membrane capacitance, r is the leak resistance of each neuron, W_ij is the synaptic efficacy between neurons j and i, and v_i is the activity (or firing rate) of neuron i that depends solely on the potential as

where σ is a monotonically increasing function of u with saturation to prevent runaway dynamics. We adopt σ(x)=11+exp(-x). Therefore, as σ(0) = 0.5, each unit has a positive output at the resting state, allowing the network to have a baseline activity without external input. We shall refer to either the vector v(t) or u(t) as “states”, which should be clear from the context.

The convergence of the flow to stable states for the case of symmetric synaptic weights W_ij has been demonstrated (Hopfield, 1984). Throughout this work, whenever we integrate these equations using Euler method, we do so until the network reaches convergence, defined as |dudt|∞<ϵ, where ϵ = 10⁻⁶.

The states v(t) of the CHN evolve on [0, 1]^N through continuous dynamics, with N the number of neurons. Any state in this space may represent a stored pattern. For simplicity, we restrict ourselves to store binary patterns for which active neurons have a high firing rate, v_i≈1, and inactive neurons have a low firing rate, v_i ≈ 0.

Hence, a pattern x^μ is defined as a binary vector such that xμ=(x1μ,x2μ,…,xNμ) where xiμ∈{0,1} for each unit i. A pattern is read from the state of the network at time t using a threshold:

Given a binary pattern x^μ, it is convenient to define target potentials u~μ as

We adopt here u_target = 6 to ensure proper pattern reading following the thresholding procedure.

Inspired by previous work on DHN (Diederich and Opper, 1987), we introduce, in Algorithm 1, a perceptron-inspired learning algorithm for efficient storage of correlated patterns in CHNs. The algorithm minimizes the error between the target states and the network's equilibrium states, ensuring that each memory becomes a stable state. Gradient descent methods typically require small step sizes to prevent the optimization process from becoming unstable and to reduce oscillations around local minima. In our implementation, we select α = 0.0001 as the learning rate to ensure stable convergence of weight updates. The derivation of the weight update rule can be found in the Appendix 1. Although a rigorous proof of convergence for the algorithm is beyond the scope of this work, we expect that arguments demonstrating the convergence of the gradient descent algorithm (GDA) in the context of DHN could be adapted for this purpose (Diederich and Opper, 1987). Here, we rely on numerical evidence showing the network's ability to successfully query and revisit stored patterns.

Following the network training with the GDA, each activity vector u~μ becomes an attractor. The network can now be queried as the system reliably converges to the nearest stored state from a partial cue. The corresponding binary patterns x~μ can then be accurately retrieved by thresholding at time t_f (Equation 3) when the system reaches convergence. The querying procedure is detailed in Algorithm 1 (Appendix 2) and illustrated in Figure 1.

It is worth emphasizing that despite the continuous nature of our model, which theoretically allows for a richer state space, we deliberately restrict ourselves to binary patterns. The interest of using a CHN is the simplicity it allows when implementing our pattern recovery mechanism (Section 2.4), limiting the appearance of false memories as spurious states, which are often encountered in DHNs (Amit et al., 1985b). A variant of the DHN with stochastic units, the SHN (Amit et al., 1985b), would be a possible candidate to implement our algorithm, as they tend to visit only the stored patterns by properly controlling the annealing temperatures. However, the stochastic properties of these networks would require a more complex setup.

While the GDA effectively enables the storage of correlated memories, its implementation in Algorithm 1 reveals a significant limitation: It requires multiple iterations over the entire set of patterns to achieve convergence. Without the ability to reprocess all patterns, the network would suffer from catastrophic forgetting, where learning a new pattern in isolation rapidly erodes previously stored memories (McCloskey and Cohen, 1989; Robins, 1995; Kirkpatrick et al., 2017; Shen et al., 2023). By repeatedly processing all patterns, the algorithm can find a weight matrix W that properly separates the patterns, despite their correlations (Diederich and Opper, 1987). Adding a new pattern, therefore, requires access to all previously stored patterns from an external source.

This requirement for external access to the complete memory dataset stands in contrast to biological learning systems, which must incorporate new information while maintaining past memories without relying on an explicit external copy of the already stored data. To overcome this external dependency and move toward more biologically plausible learning, a solution is to develop a mechanism that allows the network to internally recover its stored memories.

The development of such an autonomous retrieval mechanism would allow us to exploit the GDA's ability for the continuous incorporation of correlated memories. The continuous learning algorithm is formally defined in Algorithm 2. The act of retraining the network on the whole set is called a rehearsal. Recovering the whole set from the network to allow rehearsal is called retrieval.

In this section, we present the autonomous retrieval (AR) mechanism allowing the recovery of stored correlated patterns in a network.

Given a trained network initialized at the “neutral” state, u_i = 0 for each unit i, the network dynamics described by Equation 1 converge deterministically to a given stored attractor, which is thus “retrieved”. This attractor can be seen as the dominant attractor from the neutral state. The goal now is to allow for the exploration of other states to permit a complete recovery of the stored memories. To do so, we introduce the adaptation terms A_i.

with v_i(t_f) the firing rate of neuron i after convergence of the dynamics. β represents the adaptation strength and is chosen to be small, typically below 0.1. Adaptation terms could correspond to various components commonly observed in the mammalian central nervous system. On short time scales, spike frequency adaptation (SFA) of excitatory neurons functions as plastic self-inhibition (Ha and Cheong, 2017; Peron and Gabbiani, 2009). Repeated stimulation progressively reduces firing activity in the neuron, mirroring the dynamics produced by our adaptation A_i. Alternatively, A_i can be interpreted as recurrent inhibition mediated by local inter-neurons, which frequently exhibit Hebbian plasticity (Kodangattil et al., 2013; D'amour and Froemke, 2015).

After each visited attractor, i.e., memory retrieved, its basin of attraction is made smaller by the update of the adaptation term (Equation 5). Once a pattern has been inhibited, the probability for the network to converge into it from the neutral state is reduced. By resetting the network to the neutral state after each convergence-inhibition cycle, we allow the sequential recovery of stored patterns.

Figure 2 illustrates the sequential recovery of two stored patterns and the modification of their attractor basin during the procedure. The geometry of these attractor basins explains why a minimal inhibitory influence, resulting from a small β value, is sufficient to alter the trajectory. The neutral state resides near the separatrix that divides the attractor basins. Consequently, even a slight modification of the separatrix position, caused by inhibitory potentiation, can significantly redirect the network's trajectory.

The increase in adaptation tends to distort the stable states associated with stored patterns. As this distortion is minimal for small β, it is mainly compensated for by the thresholding mechanism used to read the network output (Section 2.2). To further reduce the impact of this distortion, we divide the convergence dynamic into two phases, the “biased” phase and the “free” phase. The biased phase guides the network to converge toward states that have not been retrieved. It corresponds to the simulation of the network with adaptation, Equation 4, until convergence. The optional free phase then allows the network to complete its convergence to an undistorted stored state. It corresponds to the simulation of the network without inhibitory synapses (Equation 1) until convergence. The whole procedure is detailed in Algorithm 3.

Figure 3 provides a visualization of AR for binary-pattern representation of handwritten digits from the MNIST dataset (LeCun et al., 1998). For the first iteration, the inhibitory drive is null as no pattern has been retrieved. The pattern corresponding to the binary picture of a 3 is recovered. The network state is reinitialized with the updated adaptation (A). The network now inhibits the recovery of a 3, which induces convergence toward a second pattern, here a 4. Adaptation is updated again and now inhibits both the 3 and the 4 together. Inhibition of the 3 and the 4 combined allows the recovery of the 5 and so on. For the recovery of MNIST binary digits, as a large inhibitory coefficient β has been chosen, the free phase is mandatory to reduce the distortion of the stored pattern.

The order in which patterns will be visited is an emerging feature of the learning algorithm that has not been studied in this work. With each iteration of the AR algorithm, inhibitory synapses undergo potentiation. This growth is constrained only by the number of iterations and the value of β. Theoretically, such an unbounded growth could cause the adaptation A_i to disrupt the attractor dynamics induced by W. However, in practice, this potential issue can be managed by adjusting the value of β based on the number of iterations. Specifically, when more iterations are required, using a smaller β is sufficient to maintain robust pattern retrieval without compromising attractor dynamics.

In Figure 4, we illustrate the dynamics of the CHN during the retrieval of stored memory patterns in networks with various loads. The load corresponds to M/N, with M the number of stored memories, and N the size of the network. For low loads (see Figure 4 top), the system converges sequentially to the attractors corresponding to the stored patterns. Once every memory has been recovered, if the simulation is not ended, the network falls back into the stored states without showing any false memories. The lower the value of β, the longer this dynamic can proceed without encountering spurious states. The free phase has no utility in this scenario, the attractors are well separated, and the disruption of the energy landscape by adaptation is minimal. For critical loads (see Figure 4 middle), the retrieval process becomes more challenging. The attractors exhibit reduced separation, and the network struggles to converge to the stored states once inhibition is applied. In this scenario, the free phase demonstrates its utility. At the end of the biased phase, during the second iteration of the AR, the network remains in a mixed state characterized by ambiguous correlations with multiple stored patterns. The free phase enables the network to resolve this ambiguity and ultimately converge to a stored state. At high loads (see Figure 4 bottom), the network successfully retrieves some stored states but mostly fall into spurious attractors. Under these conditions, even the free phase cannot rescue the recovery dynamics.

These autonomous recovery dynamics are reminiscent of memory replays observed in the central nervous system of mammals during quiescent states, such as sleep or rest. This process is believed to function as a consolidation mechanism that mitigates or modulates memory forgetting (Robins, 1995; Louie and Wilson, 2001; Peyrache et al., 2009; Fauth and Van Rossum, 2019; Tononi and Cirelli, 2014).

The use of recurrent plastic inhibitory synapses has been tested and shows the same qualitative dynamics as that observed for networks with units that undergo adaptation. The results and the model are detailed in Appendix 4. As implementing adaptation requires less computation and parameters, the following work focuses only on the adaptation model.