Since 1907 (Lapique, 1907), qualitative scientific study has been conducted on the membrane voltage of neurons. Compared to the many-variable and intricate H-H model (Hodgkin and Huxley, 1952), the integrate-and-fire (IF) neuron model and leaky-integrate-and-fire (LIF) neuron model have a significantly reduced computational demand and are commonly recognized as the simplest models among all popular neuron models while retaining biological interpretability (Burkitt, 2006). The spiking neuron model is characterized by the following differential equation (Gerstner et al., 2014):

Where u(t) represents the membrane potential of the neuron at time step t, x(t) represents the input from the presynaptic neurons, and τ is a time constant. What's more, spikes will fire if u(t) exceeds the threshold V_th. The spiking neuron models can be described explicitly iteratively to improve computational traceability.

Here, t and n, respectively, represent the indices of the time step and n-th layer, and o^j is its binary output of j-th neuron. Furthermore, w^j is the synaptic weight from j-th neuron to i-th neuron, and by altering the way that w^j is linked, we can implement convolutional layers, fully connected layers, etc. To be more precise, the spiking neurons become the IF neuron if g(x) = τ and the LIF neuron if g(x)=τe-xτ. Since h(·) represents the Heaviside function and Equation (4) is non-differentiable. The following derivatives of the surrogate function can be used for approximation.

The working schematic of spiking neurons is shown in Figure 1 (Eshraghian et al., 2021).

In Section 2.4, a transform pair for tensors are used to describe the decoding process. To better understand that process, here we first give some preliminary tensor definitions. A tensor with N dimensions is defined as P∈ℝI1×I2×⋯×IN. Elements of P are denoted as p_{i1, i2, ⋯ , iN}, where 1 ≤ i_n ≤ I_N. The n-mode unfolding vectors of tensor P are the I_n-dimensional vectors obtained from P by changing index i_n while keeping the other indices fixed. The n-mode unfolding matrix P(n)∈ℝIn×I2I3⋯In-1In+1⋯IN is defined by arranging all the n-mode vectors as the columns of the matrix (Kolda, 2006). The n-mode product of the tensor P∈ℝI1×I2×⋯×IN with the matrix B∈ℝJn×In, denoted by P×nB, is an N-dimensional tensor Q∈ℝI1×I2×⋯×Jn⋯×IN. Hence, we have the following transform pair that will be used later in the image decoding process.

Previous study (Comşa et al., 2021) has applied a TTFS encoder to encode more salient information as earlier spikes and gained good results in reconstruction tasks. This encoding method is inspired by the idea of a rapid information process with spiking data (Thorpe et al., 2001). Here rti,j is the response of a pixel of an image at time step t. Equation (7) shows the calculation of rti,j for TTFS.

In terms of rti,j, after obtaining it, spike sequences are generated using the same methods as Algorithm 1 in this paper. There are two obvious disadvantages of TTFS.

In this section, we propose the so-called Undistorted Weighted-Encoding (UWE) to encode the input images into spike sequences, as opposed to distorted encoders such as the time-to-first-spike (TTFS) encoder. Specifically, UWE can encode n-bit image [0, 2ⁿ−1] theoretically and a toy example of our coding is shown in Figure 2. In what follows, Algorithm 1 illustrates the process of encoding an image.

For simplicity, we set n = 8¹ in our work, since the inputs are 8-bit image [0, 255]. Thus, we use 8-bit UWE in this work. Especially, in Algorithm 1, x^{i, j} is a pixel in the image x. After input encoding process, x is transferred into the spiking sequences r→t∈ℝT×H×Wand rt∈ℝH×W accumulates information for each time step. Additionally, UWE is capable to be easily integrated with a neuromorphic chip which is introduced in the discussion part. This means n-bit image can be transferred into spiking sequences by the neuromorphic SAR ADC circuits (discussed in Section 4.4) without any floating arithmetic.

The decoding method that (Kamata et al., 2022) uses for reconstruction tasks is categorized as MPD. Actually, MPD is similar to our Undistorted Weighted-Decoding (UWD) to some extent. This method, like UWD, applies a weight series to encode. However, the weight values of MPD are from 2 to 0.8 and it calls a float artificial neuron (tanh function) before returning outputs. This means the n-bit decoding matrix A in UWD is adjusted to Θ = {θ^T−1, θ^T−2, ⋯ , θ⁰} and θ = 0.8, then a tanh function is used to get the real-valued reconstructed image Y^. The mechanism of UWD will be introduced in the next section. Furthermore, there is a noticeable disadvantage: MPD will induct floating arithmetic, which is unfriendly to neuromorphic chips.

To overcome the disadvantages of existing decoders, we also present an Undistorted Weighted-Decoding (UWD) to decode the output spiking sequences ô_t ∈ ℝ^H×W (t = T−1, T−2, ⋯ , 0) into the final image Y^. This decoding process is actually a symmetric process of UWE, which means UWD will transform the spiking sequences into a n-bit image. According to the preliminary, we use the output spiking sequences ô_t (t = T−1, T−2, ⋯ , 0) to build a tensor O^∈ℝT×H×W. Then, we define a n-bit decoding matrix A∈ℝ^{1 × T} = {2^T−1, 2^T−2, ⋯ , 2⁰}. Similar to UWE, we also set T = 8 in the decoding process. In Section 2.1.2, we have already introduced tensor multiplication. The final decoding process can be described by the following formula:

Where O^(3)∈ℝT×HW is the 3-mode unfolding matrix of O^ while Y^(3)∈ℝ1×HW is the 3-mode unfolding matrix of Y^∈ℝH×W×1. Hence, from the knowledge of tensor and transform pair effectively introduced in the preliminary part (Equation 6), we can get the final output image Y^. As the parallel inverse process of UWE, the decoding method can be realized by the neuromorphic chip we discussed later as well. The neuromorphic DAC circuits (discussed in Section 4.4) can convert spiking sequences to a real-valued reconstructed image without the use of floating-point arithmetic.

A previous study has applied for training high-performance SNN (Wu et al., 2018; Jin et al., 2022). Noticeably, while examining the stability of a classification task, some researchers applied STBP for image generation (Comşa et al., 2021). The standard backpropagation only considers the spatial information, which can easily be underfitted and STBP overcomes that shortage. In order to compare STBP with our Independent-Temporal Backpropagation (ITBP) in the noise-removal task, the loss function corresponding to STBP is shown below.

According to this loss function expression, the process of updating parameters is presented. To fairly compare, we show how LSTBP updates wnj in spatio-temporal domain. Other cases of updating parameters of STBP can be seen in Wu et al. (2018)'s work.

As Equation (12) shown, while STBP updates wn+1j, ot,n+1i, ot-1,n+1i, and ot,nj are all connected with wn+1j. In Figure 4, unlike STBP, error backpropagation of ITBP will not go through the decoder. Hence, ITBP is more efficient than STBP. Because error backpropagation of ITBP is in latent space but representational space for STBP. In other words, there is a risk of overfitting. Since there is a clear difference between ITBP and standard backpropagation: during the training process, ITBP only encodes the labels and does not decode network outputs; while ITBP does not encode the labels and does decode network outputs during the testing process. STBP has overcome standard backpropagation in noise removal task (Comşa et al., 2021). Later in the experiment, ITBP performed even better than STBP in a similar task.

To show the Independent-Temporal Backpropagation (ITBP) training framework, we create the loss function LITBP where the weighted mean square error is used as the error index. The expression of it is described below:

Where N is the number of training examples and ||·||_F represents the Frobenius norm, T is the total time step and we set T = 8 for our UWE and UWD. From the equation above, we regard LITBP as a function of w (weight). To obtain the derivative of LITBP to w is necessary for the gradient descent. To obtain the final ∂LITBP∂wnj, the critical step is to obtain the ∂LITBP∂ot,ni and ∂LITBP∂ut,ni at time t. Now, we show the insight of getting the complete gradient descent. First, from Equations (2) to (4), the output of spiking neurons ot,n+1i can be represented below:

Where wn+1j is the synaptic weight which links the output of n+1 layer spiking neuron ot,n+1i with the one of n layer ot,nj. According to Equations (2) to (4), we can calculate ∂LITBP∂ot,ni and ∂LITBP∂ut,ni as follows.

By Equation (5), the following derivatives of surrogate function Equation (16) can be used for approximation.

Here, ∂LITBP∂ut,ni is the intermediate variable on the step of updating parameters wnj, from Equations (14) to (16), we can solve Equation (13) as follows.

Hence, the way we update parameters will be shown below.

To state how we update weights within one epoch clearly, Algorithm 2 is shown. For Independent-Temporal Backpropagation (ITBP) in this paper, it is a non-cross-path backpropagation. That means it only propagates spatially not temporally. In other words, it is a single-modal spatial representation which means single-modality simplifies and enhances ITBP's efficiency. Last but not least, ITBP only propagates spike sequences of coded labels.

Currently, research uses leaky-integrate-and-fire (LIF) neurons for SNN, believing its more complex differential equation (Gerstner et al., 2014) can boost performance. Our experiment disproves this bias. To conduct our experiment, we use commonly used datasets (MNIST, FMNIST, and CIFAR10). In our method, the parameters of the IF model are set as V_reset = None, V_th = 0.077 in 1 × 1 convolution layer (an experience parameter corresponds to best performance), and V_th = 1.0 in all the other convolution layers. In terms of LIF neurons, τ = 1.1, and all the other parameters are set identically to IF neurons. In the majority of instances, as shown in Table 1, IF neurons usually do better than LIF neurons at this task, regardless of the noise level or dataset.

TTFS and MPD are discussed relatively in depth in the rethinking part (Sections 2.3.1 and 2.4.1) and introduction. Since they have been used to generate images (Kamata et al., 2022). They are the two most comparable methods for our UWE and UWD. Table 2 displays all results. UWE is always superior to TTFS when conducting a univariate experiment, and UWD is always superior to MPD too. In addition, the UWE-UWD combination performs exceptionally well for STBP.

Experiments demonstrate that ITBP is superior to STBP in terms of the PSNR evaluation metrics. LSTBP and LITBP are applied respectively with the same U-VTSNN architecture. Table 2 displays the outcomes of these two backpropagation techniques on the MNIST dataset with η = 0.2. All these results well proved the superiority of our ITBP.