Given the spatiotemporal characteristics of most physical stimuli, sensory information is encoded both spatially and temporally. Temporal precision seems to be critical for the accurate representation of physical sensory stimuli, which at least is shown by spike-timing-dependent plasticity (STDP). Therefore, we now focus on differences in the temporal properties of processed patterns, especially on the delays of neuronal answers to input patterns as a kind of temporal coding.

The explicit application of temporal coding to the models of pattern separation has rarely been done, although much evidence has been reported that temporal coding is useful for pattern separation. The local properties of activated neurons (spatial coding) or frequency changes (rate coding) up to date are mainly used as criteria for the separation of patterns.

In fact, there is growing evidence for temporal encoding strategies in neural networks. Indeed, there is an increasing evidence of their direct involvement in separation processes. Thus, Lisman and Jensen (2013) directly relate to the pattern separation by spiking breaks produced by gamma frequency arrest. They assume that different spatial information is represented in different gamma subcycles of a theta cycle.

Pernía-Andrade and Jonas (2014) suggest that rate coding schemes in dentate gyrus granule cells, which are ascribed to pattern separation, would be inadequate compared with temporal coding schemes. Their results are consistent with the idea that a temporal coding scheme is used in dentate gyrus granule cells.

Madar et al. (2019) provided the first experimental evidence of temporal coding strategies for pattern separation in the hippocampus. Examining the temporal properties of input and output patterns rather than differences in firing rates or firing locations, the authors showed that the suprathreshold responses of some gyrus cells were highly decorrelated compared with their inputs. The authors presented different ways to measure the similarity between spike trains and suggested that pattern separation could be achieved by multiplexed neural codes.

Our model uses spike patterns as input, generated by different numbers of neurons firing at different high gamma frequencies. Input similarity is measured by spatial overlap. As an output, we measure both the temporal and spatial properties of patterns in the output. We propose that the separation is primarily made by time to first output spike. This temporal property arises in each individual output-neuron and is used as the basis for the separation processes. Only in a further step is the temporal code extended by a spatial code through expansion recoding or by synfire chain.

In essence, our model shows that temporal encoding by time-delays to the first spike allows the individual output neurons to differentiate their input, especially when this process is assisted by rising subthreshold membrane potential oscillations (SMOs). We propose that the first step in separating spatially overlapping input patterns is achieved through temporal encoding: depending on the spatiotemporal input and the theta phase of SMOs, output neurons will fire sooner or later. Only through a second mechanism, the temporal properties of activated neurons extended by a spatial code, namely, through expansion recoding and sparse connectivity in combination with a competing inhibition process.

As an additional option for the interaction between temporal and spatial properties of patterns, a neural mechanism is used to convert temporal differences into spatial ones using a simple synfire chain model (Abeles, 1982; Ikegaya et al., 2004; Hosaka et al., 2008; Larson et al., 2010). Such models suggest a succession of end-to-end excitatory neurons (neuronal chain). An input spike propagates along the chain of neurons with synaptic delays.

Spatial pattern separation was ascribed to the dentate gyrus of the hippocampus (Berron et al., 2016). The study about temporal pattern separation from Madar et al. (2019) also examined how the input-output transformation of multiple hippocampal cell types works in the terms of pattern separation. The assignment of pattern separation to the dentate gyrus should also apply to temporal properties. The results of Pernía-Andrade and Jonas (2014) agree with this idea because dentate granule cells use a temporal coding scheme. Braganza et al. (2020) point out that in addition to the feedback, inhibition sparsity and temporal oscillations in the dentate gyrus have a critical influence on a proposed pattern separation function. However, it should be noted that pattern separation is not unique to this area of the brain. Similar phenomena also manifest in other neural circuits, e.g., the cerebellar cortex and insect mushroom body. However, recent studies from human single-neuron recordings have suggested that there may be no pattern separation in the human hippocampus (Quiroga, 2020). Others insist that experimental evidence does not rule out pattern separation in the hippocampus (Rolls, 2021; Suthana et al., 2021).

A spiking neural network (SNN) model is presented to demonstrate a prototypical approach of pattern separation. It consists of 3 neuronal ensembles: the input neurons (I-neurons), the representation neurons (R-neurons), and the time to space extension neurons (E-neurons), each accomplishing a specific task (Figure 1). Beside the input function of the ensemble of I-neurons (as shown in section “Summation in R-Neurons”), the other two ensembles solve the following tasks: spatial and temporal separation as key part of the ensemble of R-neurons and time to space transformation as an additional option of the ensemble of E-neurons. Separation is generated between I-neurons and R-neurons, extension from time to space (temporal parameters are complemented by spatial parameters) is generated between R-neurons and E-neurons.

The random connectivity between I-neurons and R-neurons is organized as follows: eight I-neurons have excitatory synapses with every six dendrites of 1–128 R-neurons, representing an expansion factor (EF) of 0.125–16. Each of the dendritic branches of R-neurons is connected by 3 synapses from I-neurons. The connections of the single dendritic branches are diluted, i.e., they receive input only from 3 to 6 I-neurons, not of all. For example, as shown in Figure 8.

For simulations, 3–6 of 8 I-neurons propagate high-gamma aligned spike trains to the 6 dendritic branches of 1–128 R-neurons. A total of 3 synapses are formed per branch. This results in 18 synapses per R-neuron from 8 randomly selected I-neurons. So first, we are using a dense connectivity of average 2.25 connections per I-neuron. Additionally, we investigate the separation effects if sparse connectivity is applied. For sparse connectivity, 3–6 of 8 I-neurons propagate the spike trains to the 6 dendritic branches of R-neurons but only one synapse is formed per branch. This results in only 6 synapses per R-neuron form 8 randomly selected I-neurons, a sparse connectivity of average 0.75 connections per I-neuron a third of the former dense connectivity.

The goal of the simulation is to show pattern separation in a prototypical way in a small SNN that combines the just described conditions.

The simulation program is written by the author (H.L.) in Python 3.7. Parameters for somatic, dendritic, synaptic, and oscillation properties are shown in Supplementary Data Sheet 1. The simulations are calculated with a time resolution of 1 ms. The values and functions of the simulation are determined as follows:

As Figure 4 shows, spatial separation clearly depends on EF. However, spatial separation effects of SP = 60% are observed even without any expansion and for EF values of less than 1, a partial spatial separation remains. For two times as many R-neurons as I-neurons (EF = 2) and the spatial SP is already about 80%. It increases to 90% and more when EF increases to 16.

Large differences between the spatial and temporal measurements of separation (Figure 4) are observed. The temporal separation along is always very high (over 90%). It is largely independent of EF. Surprisingly, even a single R-neuron separates the input temporally by about 95%, which means that different inputs (also very similar) produce different times of spikes in a single R-neuron. The repetitions of same input (averagely 5 per package) always result in identical spike times. The average spatial overlap of input patterns of about 60% is reduced to about 6% temporal overlap of output patterns in R-neurons representing SP = 90%.

Since spatial separation has mainly been studied so far, the distinction between temporal and spatial separation opens up new avenues for our understanding of the information processing in the brain. In particular, the high-temporal SP of a single neuron opens up new neuronal processes to represent similar objects as different without learning mechanisms.

Two kinds of connectivity (as described in section “Model”) are used: sparse connectivity (0.75 synapses per I-neuron) reduces the temporal SP compared with dense connectivity (2.25 synapses per I-neuron) whereas the spatial SP remains about the same.

The differences between spatial and temporal separation along EF raise the question of key influences on both measurements of separation. Expansion does not seem to affect the temporal separation. However, it increases spatial separation: a higher number of R-neurons allows the early activation of a neuron that is better connected than any other for a given input.

We then examined whether partially random connectivity from I-neurons to R-neurons is essential for temporal separation. Complete connectivity is implemented by linking the 3 dendritic branches of a single R-neuron through all 8 I-neurons. We then examined whether partially random connectivity from I-neurons to R-neurons is essential for temporal separation. SP is calculated in the same way as previous simulations. Full connectivity as well as incomplete random connectivity result in the same temporal SP = 93%. All identic input patters result in identical output.

In addition, it is sometimes reported that sparse connectivity is conducive to the pattern separation. So we simulated our model with sparse connectivity. Sparse connectivity boosts temporal but not spatial SP. However, if “sparseness” is defined as “reducing the fraction of active neurons, that make up a “sparse” population code” (Cayco-Gajic and Silver, 2019), then sparseness is also a very important feature for separation in our model. In particular, backward inhibition significantly reduces activity above the threshold of R-neurons and produces a sparse population code in all simulations of our model.

Our results show that neither EF nor incomplete random, but also not sparse connectivity are key parameters for temporal separation. However, EF and incomplete random connectivity are prerequisites for spatial separation.

To study the influence of input parameters on the summation over time, we independently varied the number of activated I-neurons and their firing frequencies. Their individual SP was measured.

By increasing the number of activated I-neurons, the time to the first spiking of R-neurons is reduced. The output times, therefore represent the strength of the input in an inverse manner. This relationship is a consequence of the summation of the input spikes and is accompanied by an increase in potential by SMO. However, when the I-neurons fire at different frequencies, this strong relationship is reduced. Therefore we simulated our model with full connectivity and with the same frequencies of 90 Hz for all I-neurons. If the connections between I- and R-neurons were random, the relationship between output times and input strength would also be affected. A high correlation would only be given by a statistical mean of as many output neurons as possible. With full connectivity, different but fixed output times (between 11 and 113 ms) are generated by a single R-neuron. Each number (1–8) of the activated I-neurons produces a different output time. This was also true for the average output times when using 128 R-neurons with random connectivity.

The output times are transferred to the firing of various E-neurons with the help of the synfire chain arrangement. Our model allows counting the number of activated I-neurons by temporal separation. Each number of activated I-neurons is represented by a specific E-neuron.

By increasing the frequency of activated I-neurons, the time to the first spiking of R-neurons is shortened. The output times should therefore correspond to the input frequency. To check this, all I-neurons fire at the same frequency and this frequency is systematically varied. We simulated our model with full connectivity and with frequencies of I-neurons between 50 and 150 Hz with an interval of 10 Hz. In this arrangement, different output times (between 23 and 89 ms) are calculated from a single R-neuron in relation to the spiking frequencies of three I-neurons. Using the synfire chain arrangement, different frequencies of I-neurons lead to the spiking of different E-neurons as the output. The input frequency is represented by specific E-neurons. Our model allows the representation of frequencies by temporal separation.

In a further step, an attempt is made to simulate a simple visual object as input using 8 I-neurons that fire at different frequencies. With this we want to find out whether our model also represents objects separately with the input parameter for objects used there. We assume that different properties of environmental objects are represented by the activities of a few I-neurons. As the properties of simplified objects, 3 possible shapes, 3 possible colors, and 2 possible sizes are used. Each object is defined by 3 properties, one for shape, one for color, and one for size. The activation of I-neurons 1, 2, or 3 represents 3 possible forms of the object, the activation of neuron 4, 5, or 6 represents 3 possible colors, and the activation of neuron 7 or 8 represents 2 sizes of the object.

By combining the three properties, our model aims to produce well-separated representations of 16 different objects. Examples of two objects given activated I-neurons as input are shown here:

Separation power is tested by presenting all 18 possible objects (plus 2 randomly chosen repeats), each represented by 3 activated I-neurons in our model. The 20 objects are presented in the form of 20 packages, each with a new random connectivity. To reduce the average times in the output from the end of the time domain (caused by activation of only 3 I-neurons) toward its center, the synaptic weights from I-neurons to R-neurons are doubled (from 2 to 4). The SP is measured in two ways: first, as a percentage of correct repetitions of output patterns when the same objects are presented. This criterion by Madar et al. (2019) is named as “input reliability” that is the ability of a neuron to reproduce the same output pattern on the repetitions of input pattern. Second: as a percentage of different output times of the single R-neuron in relation to the number of 18 different input objects. The transformation of spiking times of the R-neurons into spatially different E-neurons is accomplished with the help of a “synfire wave.”

The first result: input reliability is perfect. Approximately 100% of repetitions are represented by equal output times of R-neurons and equal E-neurons.

The second result: the average number of different spike times for 18 different objects is 14. This means that 78% of the different objects are represented by different output times (and subsequently also by different E-neurons). It should be noted that 77% of correct object identification is generated without any learning mechanisms and is based on the spiking times of a single R-neuron. While the spatial overlap of the 18 different input objects is 35% and the spatial overlap of the output patterns of a single R-neuron numbers only to 3.6%. Therefore, similar to previous results the mean temporal SP is 90%.

The separation effects shown so far do not require any learning mechanisms. They are generated by different excitation with constant synaptic weights. In particular, hsLTD has previously been proposed to facilitate spatial separation effects. This is confirmed in our model (Figure 5). However, this promoting effect of hsLTD is only observed for spatial separation with EF > 1. The temporal separation does not seem to be affected by hsLTD. For EFs of 2 and more, the spatial measurements of SP are over 90%.

Input patterns have produced a time-separated output in R-neurons. In our model the R-neurons temporal output is perfectly converted into a spatial one using “traveling EPSPs” by a synfire chain circuit as described above (as shown in section “Synfire Chain” and Figure 6). Depending on the output time, a different E-Neuron fires. The temporal overlap corresponds exactly to the spatial overlap (Figure 7). There must be as many output neurons available as different input times can occur (about 100 in our simulations).

The simulations of our model provide the following results for pattern separation:

The model is reasonably robust to variations in the gamma and theta frequencies used: SMO-theta frequencies of 3 HZ in R-neurons (instead of 5 HZ) and gamma frequencies of the input between 60 and 130 HZ (instead of 90–160 Hz) lead to a reduction in the SP of only about 15%. Both high and low theta oscillations in the human hippocampus have recently been identified by Goyal et al. (2020).

Information processing through delay times is not entirely new. Henze et al. (2002) proposed that the information in single granule cells is converted into a time delay code by CA3 pyramidal cells and interneurons. Higher granule cell spike frequencies produce shorter delays. A similar magnitude of activation temporally discharges CA3 targets together, thereby increasing their connectivity to one another. In contrast, our model uses the different distribution of spike times. First spikes in R-neurons are distributed in time and different delays separate the spatially overlapping input patterns. Sufficient summation effects are required to differentiate the input patterns. Temporal summation can be improved by longer EPSPs. Indeed, higher EPSP decay time constants were found in the hippocampus, where separation processes are likely. The decay time constant in dentate gyrus granule cells is about 6 ms and in hippocampal CA3 neurons is 11 ms (Kowalski et al., 2016). We use a decay time constant of 7 ms for R-neurons, for the other neurons in our model we use 3.5 ms.

According to our model, EPSPs by input patterns arrive the R-neurons during the ascending part of theta SMOs. Due to this temporal position of the input in relation to the theta phase, the lower cumulative values of membrane potential can also become overthreshold due to the increasing oscillation.

Theta oscillations are present in all subregions of the hippocampus and in the granule cells of the dentate gyrus. Increased activity patterns in the target cells of the dentate gyrus or the CA3 region due to input from the entorhinal cortex in the ascending phase of theta waves probably could be indicative of the proposed separation mechanisms. In fact, the overall population of active pyramidal cells fires the most action potentials per theta cycle just after theta cycle bottom (Mizuseky et al., 2009). However, this might be oversimplified to validate our model. Theta oscillations can be generated intrinsically (e.g., Fellous and Sejnowski, 2000 in the CA3 field) or extrinsically by synaptic circuits (e.g., by rhythmic perisomatic inhibition). The physiological mechanisms that evoke theta field potential and the temporal coordination of individual neurons across anatomically sequential subregions through theta oscillations are not yet fully understood. Population activity in hippocampal subregions does not merely reflect their input but also represents the result of autonomous local computation. Mizuseky et al. (2009) found an offset by a half-theta cycle of downstream dendritic excitation of CA3- and dentate gyrus neurons although the peak of population activity in the upstream entorhinal cortex structure corresponded well with the timing of dendritic excitation. Theta dynamic seems to allow for a considerable degree of independence of local circuit computation in the successive stages of the EC-hippocampal system.

It has always been assumed that expansion recoding plays a key role in pattern separation (Marr, 1971). The expansion can be quantified by the ratio of the size of the input population to the population of the expanded layer. In complete contrast to this, our simulations show that at least high temporal separation can occur through individual neurons and even taking place with a reduction. Spatiotemporal patterns are in a first step separated in the millisecond range of a rising theta wave (∼100 ms). Different spike times replace the commonly used different neurons. Only for the transfer of temporal into spatial separation a moderate expansion is required as a second step, which can be done by different mechanisms, e.g., by synfire waves or by competing WTA with random diluted connectivity. Thus by an expansion factor of 2 a spatial SP of 90% is achieved if supported by hsLTD. That is consistent with Cayco-Gajic and Silver (2019), who suggested that different structural and functional properties can cause pattern separation.

In the following we try hypothetically to locate our model in the circuits between entorhinal cortex, dentate gyrus, and hippocampal CA3-region, where significant separation processes are thought to be processed. For this purpose, our model is expanded in a modular way: 8 I-neurons together with 16 R-neurons form a unit that is combined to form a larger system of n modules. Every module processes a pool of I-neurons to the R-neurons. R-neurons are trained to be equivalent to the CA3 pyramidal neurons. Their input comes mainly from the perforant path of the entorhinal cortex, from mossy fibers of dentate gyrus, and from CA3 pyramidal neurons themselves by recurrent axons. The entorhinal input is relatively weak rarely exceeding the spiking threshold of pyramidal neurons by itself. The dentate input is arrived by a small number of strong synapses (mossy terminals synapse with 11–15 different CA3 pyramidal cells, Acsády et al., 1998) and does not exceed the spiking threshold on its own. Only the combination of both inputs is able to trigger spikes in CA3 cells. Whether this happen depends on the random connectivity of both inputs. In any case, the random and sparse connectivity from the dentate gyrus limits the field of possible activated CA3 neurons. It determines the effective factor of expansion by the input from the entorhinal cortex. Only in CA3 neurons receiving input from the dentate gyrus, the input from the EC is able to become suprathreshold and to represent the entorhinal input via separation. Subsequently, the activation of individual pyramidal cells leads to autoassociative feedback circuits that can generate memories and pattern completion of the input.

Our separation model works within a single theta phase of SMO of about 100 ms. New theta phases produce new separated representations of the next input. Separation can take place sufficiently rapidly to be complete within one theta cycle.

High temporal SP increase favors the stochastic activity of neurons during information processing, resulting in an increase in entropy, which forms a correlation of perception as hypothesized by Gupta and Bahmer (2019). Note that an increase in SP will lead to a decrease in the probability of joint activity of pairs of neurons if separation is also accompanied by the control of the pairs of neurons by different sets of influences. The decrease in the probability of joint activity will increase the entropy and decrease the mutual information. However, note that the increase in entropy or surprisal information if combined with an increase in mutual information serves as key bases of perception. Thus, the separation allows new time windows for an increase in mutual information given only the presence of specific stimuli. Therefore, the proposed model provides an important mechanism for an increase in entropy during information processing underlying the cognitive functions of the brain.

We are surprised at the high temporal separation by a single neuron. Nevertheless, temporal differences in the representation patterns after separation are sometimes only one or a few milliseconds. The question arises as to how such small temporal differences can be further processed in a meaningful way. Using synfire chain processes, we make a new proposal how temporal patterns can be extended by neurons to spatial patterns for further processing. To do this, we use systematically altered excitability between neighboring neurons. The temporal separation by single neurons can be perfectly converted back into an additional spatial separation. A very different way of processing small differences in time is expansion, which was seen as central to the separation process. Incomplete random connectivity spreads the times to the first spike across a spatially extended ensemble of neurons. A competing WTA-mechanisms selects the neurons with the shortest delay. There are probably other ways of converting temporal differences into spatial ones.

We hypothesize that additional determinants not considered here, such as the diversity of synaptic weights on dendritic branches can influence the SP. Thus Rolls (2016) noted that pattern separation could be produced by a fully connected competitive net without learning in which the synaptic weights are set to random values.