The simplified structure in Figure 2A depicts how an input signal to a random network is traveling along topological stages (layers) within this network and how the network “reacts” to the input by adapting weights and activities. For this we simulate the behavior of such a feed-forward network with one neuron per stage (c.f. Figures 1A and 3) given a noisy input to layer one (activity of red neuron). As shown before (Figure 1), this activity leads to a strong post-synaptic synapse resulting in high (but not as high as red) activity at layer 1 (green neuron). This behavior is propagated until the activity vanishes in layer 4 (gray). This is the basic property of the above described input trace (cell assembly). Even if the input is reduced for a certain duration the network can re-adapt to its previous stable state very fast (c.f. Figure 3). Thus, even with a noisy external input, the network can construct a stable cell assembly along several stages.

In the following we analyze analytically the fixed points of weights and activities for each stage dependent on feed-forward connections and the input. We consider an input of average strength ℑ given to a neuron j⁽⁰⁾ = a of the first layer k = 0 (pink neuron in Figure 2A), which leads then to the input activity vj=a(k=0)=va(0)=ℑ of the first layer (the other neurons in layer k = 0 have vj≠a(0)=0). Next, we calculate the weights ωi,j(1) and activities vi(1) of a neuron i⁽¹⁾ of the next layer k = 1 assuming that the dynamics have stabilized:

Given this result, equation (1) and uj(m)=vj(m-1) the dynamics for weights in the second layer write

This equation still has only one fixed point in the excitatory regime (i.e., ω > 0):

Substituting this solution into equation (4) gives

We write v(⋅)(m) as this solution applies for all i⁽¹⁾∈{1, …, N₁} of layer k = 1 because neuron j⁽⁰⁾ = a is connected to all neurons of layer k = 1.

All neurons of layer k = 1 have the same activity and since we use an all-to-all connectivity between each layer, neurons within each subsequent layer k ≥ 1 will have equal activity, too. Hence, the (equal) activity of each neuron in layer k = m and also the (equal) strength of each weight projecting to layer k = m merely depends on the (equal) activity of each neuron in layer k = m − 1 and the number of neurons N_(m−1) in this layer:

In the next subsection we will analyze and discuss the dynamics of these equations for different parameters given the same number of neurons in each layer (N = N_k, ∀k∈{0, …, M − 1}).

Equations (8) and (9) describe the dynamics of weights and activities in a feed-forward network that is stimulated by just one input. When we look at the propagation of the activity (after stabilization of the weights) for different parameter values, we will find two qualitatively different scenarios: Activities will increase (divergent regime) or decrease (bounded regime) from layer to layer. As divergence should be avoided and bounded development enforced, the following relation has to be fulfilled for all neurons m>1:v(⋅)(m)≤v(⋅)(m-1). We now calculate lower and upper bounds for the activity and the weights so that we stay in the bounded regime.

The first inequality applies for negative firing rates. Thus, we ignore this result in the following. The second inequality defines an upper and lower bound on the activity or rather on the input ℑ:

The two constraints equations (10) and (11), and the nullcline for the weights (equation A1 in Appendix) define the weight-activity phase space (cf. Figures 4A–C) as well as the development of the fixed points of activity and weight along layers or rather stages m (cf. Figures 4D–E) in a feed-forward network. The trajectory depends on the terminal firing rate v_T and the number of neurons N (c.f. Figure 4).

The phase space consists of two qualitatively different curves. The first class of curves represents the fixed points of each layer (equation 8). These fixed points do not depend on layer number k directly but on the input to this layer which is the output of the preceding layer vim-1. In Figures 4A–C these fixed points are represented by the solid line. The second class of curves deals with the layer to layer fixed points of our system, i.e., with the previously calculated two constraints equations (10) and (11) and, additionally, equation (A1) for the weights. These constraints define the limits of a region within which activity stays bounded from layer to layer. In Figures 4A–C these curves are dashed lines (vertical black lines are v^min and v^max; the purple line is the nullcline ω^equ for ω^m = ω^m−1). The constraints v^min and v^max divide the phase space into three qualitatively different regimes: if the activity of only a single layer is smaller than v^min, the activity of the next layer “jumps” to v^min and stays there for the descendant layers. If the activity is larger than v^max, activities increases from layer to layer; only if the activity stays between v^min and v^max, activities decrease from layer to layer and stay bounded (c.f. arrows in Figure 4A). Only v^max is significantly influenced by N leading to a smaller stable range as more neurons excite one neuron in the next stage.

Now we can understand how fixed points develop from layer to layer and, thus, how large a cell assembly, given input strength ℑ, gets.

First, we apply an input from the bounded regime that causes activities and weights of the first layer to converge to their fixed points which lie on the blue line (e.g., k = 0 in Figure 4A). The resulting activity v⁽⁰⁾* is then transmitted to the next layer and causes in turn weights of the next layer to converge. Intuitively, this is visualized by a line going downward until it reaches the weight nullcline ω^equ. From here, weights together with the activity of the previous layer once again cause the activity of the next layer (k = 1) to converge, too. This is visualized by a line going leftwards to the blue fixed point curve.

This process repeats from layer to layer until the global fixed point at (v^(g) = v^min, ω^(g) = 1) is reached for layer k = g. Activities and weights will not change for subsequent layers k > g. In other words, the external input cannot influence layers k > g.

Thus, in a pure feed-forward network any kind of information storage can only exists for the first k layers since the system reaches its global fixed point for layers beyond k independent of the initial input (as long as the input is in the bounded regime).

To assess the capacity of information storage (size of input trace), we need to determine the influence of network parameters on g. It turns out that most influence on g is caused by v_T. Figures 4D,E demonstrate activity and weight development for three different values of v_T. Note that although the development moves along discrete layers, we plot continuous lines.

Next, we estimate the “number of jumps” from the distance between weight nullcline w^equ and the blue fixed point curve (c.f. arrows between blue and purple line in Figure 4A). A smaller distance (compare Figures 4B,C) leads to smaller “jumps” from layer to layer which in turn leads to more layers toward the global fixed point. However, at the same time the difference between v^min and v^max decreases which leads to a reduced total distance between the input value and v^min.

In Figures 4A1–A6 we also show numerical results of activity and weight development when the input is chosen to be in one of each of the three different regimes; this confirms our analytical observations. For inputs greater than v^max the network diverges (Figures 4A5 and A6). For inputs less than v^min the network converges quickly to its global fixed point (v = v_min, ω = 1; Figures 4A1 and A2). Only for inputs between v^min and v^max (Figures 4A3 and A4) we notice distinctively different fixed points along several layers until at around layer eight the global fixed point is eventually reached. Thus, such a feed-forward network with inputs between v^min and v^max stores different values for activities and weights, here up to layer 7. The numerical results of A3 and A4 are qualitatively very similar to the analytical curves in Figures 4D,E.

We now define the theoretical limits of this system. On the one hand we see from Figure 4D that in order to maximize g the distance between v^max (equation 10) and v^min (equation 11) should be small and v_T should then be ξ = (4κN²)⁻¹:

Given an input between v^min and v^max, the system would need m → ∞ to reach v^min. However, with v_T → (4κN²)⁻¹ the interval (v^min, v^max) from which inputs can be chosen becomes smaller – eventually zero – and the network is specialized to “store” only a few (or one) distinct inputs. Additionally, the difference between input representation (minimum fixed point weight ω^min) and global fixed point (ω^(g) = 1) also becomes smaller (green curve in Figures 4D,E). So a noisy read-out of the stored inputs is complicated as an input difference could easily fall inside the noise level. On the other hand, to get a stable read-out and at the same time to “store” a large range of different inputs, the difference between v^min and v^max should be maximized. From equations (10) and (11) we find that v_T should be as negative as possible, however, in a pure feed-forward network v_T < 0 would lead to a negative firing rate for v^(g) = v^min < 0 which is mathematically possible but biologically implausible. Thus, v_T should minimally go to zero:

To sum up, there is a trade-off, which depends on v_T, between the number of layers representing an input and the stability of this representation against noise. On the one hand, a high v_T ensures an extended representation over many layers, but, on the other hand, this representation is now quite susceptible to noise during the read-out process as fixed points are close to each other, and vice versa for a small v_T.

Signals and therefore information in feed-forward networks are transmitted from layer to layer. If we demand that there is still information of our initial signal even after many layers, the network has to maintain the spatio-temporal integrity of inputs. Ideally, inputs should not disperse randomly both in space and in time.

We therefore test the capability of a feed-forward network (Figure 2C) for signal representation and transportation. The synapses of the network are modified by the SPaSS rule. Constant weights represent the control (to mimic, for instance, weight hard-bounds). In the first of our two scenarios we present a spatially restricted signal (transport signal) to the first layer and then measure the dispersion along layers (Figures 5A,B). In the second scenario we inject as an input two adjacent signals with a small spatial gap between them (resolution signal; Figures 5C,D). With both scenarios we are able to analyze the network’s capability to differentiate its inputs after a certain number of layers. Different from the previously analyzed network the feed-forward network is constructed as follows: each neuron projects and in turn receives synapses to and from a neuron at the same position and its two neighbors in the next and the former layer.

If the weights are set to a constant value ω_c, the input smears out with every subsequent layer. This is best visible in Figure 5B when comparing the activity profile of each layer with the dashed lines of the original signal. Additionally, the amplitude of the signal gets reduced with every subsequent layer. Although smearing is reduced by using smaller constant weights ω_c, this also leads to a stronger reduction of amplitude. The SPaSS rule, on the other hand, can reduce (after weight stabilization) both smearing and amplitude reduction (cf. Figure 5A) at the same time. It is interesting to note that only in the first layer the activity of the border neurons is significantly increased. In all subsequent layers, the signal amplitude of these border neurons does not increase but merely follows the general reduction of signal amplitude.

In the second scenario smearing reduces the gap between the two initially separate signals from layer to layer (Figure 5D). For constant weights ω_c the signals are difficult to distinguish after the third layer (Figure 5D). For plastic synapses the resolution decreases, too, but both signals are still distinguishable until the signal vanishes at layer five (Figure 5C). This effect has its origin in the scaling mechanism which tries to balance incoming signals. This leads to a suppression of later signals while at the same time the strong forward pathways are being maintained.

In summary, the SPaSS rule avoids a spreading or rather smearing of spatially restricted signals in deep layers of feed-forward networks. Therefore, the rule ensures that spatially distinct signals presented to a network are still distinguishable after passing several neuronal layers.

In this section we consider step by step our feed-forward network with -i- lateral excitatory connections within layers (top row in Figure 6), -ii- feed-forward inhibition (middle row in Figure 6), and -iii- feedback (self-)inhibition (bottom row in Figure 6) while all intra-layer connections stay constant. Under these constraints, analytical results can still be obtained and we state only the main equations here. All calculations – similar to those shown above – are provided in the Appendix. In principle we could also do analytics for mixtures of the structures presented in Figure 6 but calculations get lengthy and opaque then, and the same is true for plastic intra-layer connections.

The simplest recurrent network is a single neuron with an input ℑ and a self-connecting synapse with weight ω (Figure 2B). The activity v_t at time t of such a neuron is simply the sum of a constant external input ℑ and the weighted output of the neuron a time step before. (Instead of one time step a fixed delay term d could be used.)

To obtain stability both variables v and ω have to be constant over time, i.e., ω. = 0 and v_t = v_t−1. This leads to the following dependency between activity and weight:

We insert this equation into the differential equation for weight development:

and the solutions of equation (19) are

with

These solutions have to be real (complex part equals zero) and the derivative of equation (19) at these solutions has to be smaller than zero in order to stabilize the dynamics, so parameters ℑ, κ, and v_T have to be restricted. The calculations for assessing these parameters are complex and closed-forms can not be derived anymore. Therefore, the stability for a given set of parameters has to be assessed by numerical calculations of the roots. To reduce the number of parameters only the ratio of the rates (e.g., equations 8–22) κ = μ/γ is considered and we plot the maximally allowed input ℑ^max that still leads to a stable weight. All smaller inputs (0 < ℑ < ℑ^max) lead to a stable weight ω, too, and inputs above this value (ℑ > ℑ^max) lead to divergent weight dynamics. Thus, we show the parameters which lead to stable dynamics in a κ-v_T-plot with ℑ^max color coded (Figure 7A). The self-connected neuron stabilizes best if γ is as large as μ (κ ≈ 1) but for the more realistic ratios of one order of magnitude difference (κ ≈ 10) stabilization is still possible for a wide regime. Furthermore, stability requires that v_T is small or even negative. Here we stress that (1) negative v_T does not imply negative neuronal firing rates as these systems balance plasticity and scaling (vi*≠vT; see equation 3 and Tetzlaff et al., 2011) and (2) negative v_T values can be avoided by simply adding inhibition to any of these recurrent networks (Figures 2B–D). As shown above, constant weight inhibition (Figures 6D–I) leads to a shift of the network properties toward larger allowed positive values. The general disk-like shape of the stability plot, however, does not change with inhibition added and we will thus continue to consider only excitatory recurrent networks (although v_T might be negative).

We have confirmed our numerical calculations with simulations up to four places after the decimal point for weight ω and activity v. (For example, for κ = 2, ℑ = 0.065, and v_T = 0.01 the weight and the activity in the simulation as well as in the numerics is ω = 0.5674 and v = 0.1503 respectively).

We now compare the activity of a neuron with a self-connection v_sc with the activity v_ff of a neuron in a feed-forward structure receiving the same input ℑ if equation (19) is simplified in the following way (see also Appendix): as ω is smaller than one, weight-dependent terms of order three (Θ(ω³)) and higher can be ignored. This leads to following neuronal activity:

which is larger than v_ff = ℑ.

In summary, self-connected structures in a neuronal network will raise the risk of instability. As this instability depends on the input (or rather the activity given to the self-connected neuron) this risk will decrease if a self-connected structure is in deeper layers (because activity has already declined there). However, as the resulting fixed point activity of self-connected neurons is larger than the activity of a neuron in a feed-forward structure, self-connections will extend input trace into deeper layers.

In recurrent networks there are also other types of recurrent structures than simple self-connected neurons. Thus, in the following we derive the influence of larger recurrences on the stability and activity of neuronal networks. Equations (A19) and (A20) in Appendix show that also for the bi-directional connection the activity is increased compared to the feed-forward networks. This effect could be expected as already constant recurrences enhance network activity, but the SPaSS rule sustains the enhancement and still guarantees stable weights. Interestingly, this effect does not depend on the position of the bi-directional structure within the feed-forward network. The second neuron can be part of the feed-forward network but it can also be outside of the layered structure.

Additionally, we numerically calculated the stable regime for different parameter values and plot the results similar to the self-connected neuron. This analysis (Figure 7B) confirms the result that stability is independent of the chosen parameters v_T and κ. However, we observe that the maximally allowed input ℑ^max that still leads to convergence increases compared to the self-connected neuron (mind the changed color scale bars in Figures 7A,B).

For the three-neuron ring an enhancement of these observations is observed. Its equations (equation A23 in Appendix) again show that for the three-ring the activity is increased, but less, as compared to the feed-forward networks.

Furthermore, we find that the stability range for the three-ring (Figure 7C) looks similar to the one discussed above (Figures 7A,B) and the maximally allowed input ℑ^max has further increased.

These considerations can be generalized in a straight-forward way for an N-ring, too (not shown).

To conclude, the results presented above confirm what we stated at the beginning of this section. First, if the smallest existing ring structure in a randomly wired recurrent network is stable, then every longer ring will be stable, too. Second, there is an ordering of the resulting activities (see Appendix):

(self − connected > bi − directional > N − ring > feed − forward). Furthermore (similar to the direct self-connection discussed above) if one neuron connects recurrently via a ring to itself, its fixed point activity is enlarged. Thus, given a feed-forward network, this fact ensures that also all following (down stream) layers will have an increased activity and, therefore, more layers are needed to reach the global fixed point. This effect is in general even stronger with the SPaSS rule than with constant weights (c.f. section 5) as the denominator of the activity depends on the input ℑ. Thus, the length of the input trace is increased with any excitatory recurrent structure.