Frequency modulated continuous wave (FMCW) radar is massively used in cars for advanced driver assistance system (ADAS), and due to its robustness, it is considered an automotive industry standard. As its modulated waveform, it uses a continuous monotonic chirp, whose frequency increases (or decreases) linearly along its duration. Figure 2 shows a general block diagram of the FMCW radar, in which the reference signal (Tx) is generated in the ramp synthesizer, and transmitted via the antenna array (Tx1, Tx2, and Tx3) after its radiated power is increased using a power amplifier (PA). Each receiver block (Rx) mixes the Tx signal with the amplified target echo at the output of the low noise amplifier (LNA), and creates the intermediate frequency (IF) signal, which is digitized through the analog-to-digital converter (ADC). Considering a radar echo from a single object, the received frequency ramp will have a time shift Δt proportional to the distance d to the radar sensor, which is equivalent to a frequency shift Δf, as shown in Figure 2. After down-mixing the two signals, the reflection from a single radar object will contribute a sinusoid of frequency Δf to the IF signal. This frequency is defined by: (1)Δf=2dBc0Tc, where B is the bandwidth of the chirp, T_c the chirp duration, and c₀ the speed of light. In practice, the IF signal is a superposition of reflections from multiple targets with different Δf and noise. The range of the targets can be extracted via the range-FFT (signal processing described in the Section 2.2).

Within a so-called radar frame, multiple of these fast chirps are transmitted successively to obtain the relative velocity: For an object that moves away from (toward) the sensor, the frequency shift Δf increases (decreases) between chirps, although the shift is typically so small that it cannot be recognized after the range-FFT. Yet, the phase difference ω_v of the IF signal components between two consecutive chirps (cf. Figure 2B) contains the information about the relative velocity v: (2)v=λωv4πTc,diff, with the carrier wavelength λ (3.9 mm for 77 GHz radar) and the time between chirps T_c,diff. To achieve a high accuracy for the velocity estimation, it is typically extracted by applying the so-called Doppler-FFT over all chirps within a frame (see, e.g., Patole et al., 2017 for further details).

In order to retrieve the angle of arrival (AoA) θ for one target, at least two receivers are needed. For an antenna array of two elements with a separation distance d, the reflected signal from the single target is captured with a phase difference (ω_θ). Using far field approximation this phase difference can be calculated as (3)ωθ=2πλdsinθ. By adding more receive and transmit antennas, the angular resolution for detecting target reflections and distinguishing them from other reflections can be increased. Typical automotive radar sensors have 3 transmitters and 4 receivers. While the receivers are arranged along the horizontal axis, the transmitters are arranged in an L-shape to also obtain an elevation angle (Sun et al., 2020). Hence, the so-called virtual antenna array in azimuth direction has 8 antennas. Yet, there is a trend to high-resolution radars with 64 antenna elements and above (Bilik et al., 2018; Och et al., 2018; Sun et al., 2020). The drawback of this MIMO approach is that it needs modulation schemes to ensure the separation of the individual contributions from each transmitter. The most used modulation scheme is time-division multiplexing (TDM), in which only one transmitter is enabled concurrently, but there are other approaches that use phase codes or frequency division multiplexing (Roos et al., 2019).

In the following, we describe the steps for processing a single radar frame recorded with a MIMO FMCW sensor. The IF data recorded in a frame is organized as a data cube with 3 dimensions: the number of receivers N_RX, the number of chirps per receiver N_chirps, and the number of ADC samples per chirp N_samples. A single sample is typically an integer value with 12 to 16 bits, or a complex number with two 16 bit integers in case of an IQ-baseband architecture (Ginsburg et al., 2018). Typical numbers for the three dimensions could be 4 receivers, 64 chirps, and 512 samples. In total, the complete raw data of one frame can require up to 256 KiB for the considered case of real-valued samples. The digital signal processing steps are illustrated in Figure 3, which are briefly described in the next sections. For further details see Patole et al. (2017) or Gamba (2020).

The recently published CARRADA dataset (Ouaknine et al., 2020) is one out of few publicly available automotive datasets containing not only vision and LIDAR/depth information but also radar data. Most datasets do not include radar data at all (Geiger et al., 2013; Yu et al., 2020), but even if they do, the radar data included is usually in form of point cloud information (Caesar et al., 2019; Meyer and Kuschk, 2019a; Schumann et al., 2021), providing the (x, y, z) coordinates and the relative velocity of objects. The CARRADA dataset, on the other hand, includes the range-Doppler as well as the range-angle map for each scan. Still, it is limited in size, complexity and variety compared to the aforementioned datasets, as it is recorded on a remote test track in Canada with low environmental noise.

The CARRADA dataset consists of 30 separate sequences with a mean number of 422 frames per sequence (0.7 min) gathered from a synchronized setup composed of an FMCW radar and a camera mounted on a stationary car. Out of the total 12666 frames taken, 7,193 are annotated, containing one or two moving objects (car, pedestrian or cyclist). Each frame contains 3 different annotations (bounding boxes, sparse points and dense masks), making the dataset suitable for different tasks like object detection, semantic segmentation or tracking. The experiments presented in Section 4 make use of this dataset.

The Fourier transform is typically applied in three different dimensions in automotive radar applications, i.e., the range, angle, and velocity. While the efficiency of the FFT algorithm is unquestionable, we consider SNNs for frequency spectrum analysis as they might be implemented very efficiently on neuromorphic hardware: We first discuss the use of resonate & fire (RF) neurons, continue with a recent spiking realization of the discrete Fourier transform and conclude with other brain-inspired approaches.

In addition to replacing the angle FFT with a spiking neural network, we discuss other approaches for angle calculation: Looking at the brain, this problem resembles the sound localization which uses interaural time differences (ITD) for the AoA computation. Highly experienced echo-locators such as bats employ interaural level differences (ILD) instead, which in contrast to the ITD using their small heads, allows them to capture a wide diversity of target cross-sections at different ranges by sensing pressure differences across their ears. Engineering ITD methods require the concept of phase locking and delay lines so that certain neurons show a high firing rate when a certain frequency arrives at a certain AoA (Carr and Konishi, 1990). The concept has been proven in neuromorphic hardware with spiking neurons (Pfeil et al., 2013). However, it seems challenging or even unrealistic to apply the ITD or ILD methods to radar processing: For the continuous wave radar approach, there are no time differences measurable at different receivers, also the phase shifts are very small and would need to be pre-processed to act as an input to a neural network based on ITD or ILD. More complexity is added as there is not a single transmitter, but there are multiple that alternate in being active such that input data would need to be buffered before being processed as a larger virtual receiver array.

As conclusion of our analysis, the spiking Fourier transform from Section 3.1.2 seems to be the only suitable approach for the angle-of-arrival calculation so far. Yet, further research should be carried out on replacing high-resolution algorithms such as MUSIC or ESPRIT.

The classical approach uses the constant false alarm rate algorithm to adjust a local threshold to distinguish radar object reflections from noise. In a second step, the reflections are assigned or grouped to clusters representing the same radar object. We present two constructed SNNs implementing two different CFAR algorithms and briefly discuss spiking network approaches for clustering and grouping.

The state-of-the-art approaches for target classification mostly use ANNs like CNNs or RNNs (Section 2.2.4). As mentioned in Section 2.3.3, SNNs for image and sequence classification can be obtained by conversion from DNNs or direct training. In the following, we focus on the classification of single radar objects with SNNs, i.e., we expect that only a single target is present in the input data. This can be achieved by extracting ROIs from the radar data making use of the clustered object reflections from the previous processing step.

To the best of our knowledge, so far SNNs have not been applied to automotive radar object classification, yet there is a variety of work on radar gesture recognition using SNNs which differ in the coding of the input data and network architectures: For the SoLi dataset (Wang et al., 2016), which provides sequences of range-Doppler maps, Yin et al. (2021) trained a network of several recurrent SNN layers with adaptive spiking neurons using surrogate gradients and BPTT. Similarly, Safa et al. (2021a,b) trained a spiking convolutional network achieving a higher accuracy. Both approaches turn the range-Doppler maps to spikes by thresholding. Instead, Tsang et al. (2021) feeds the spiketrains into a liquid state machine, a recurrent network of spiking neurons retaining a memory of received input, and evaluates various classifiers as read-out: Using an SVM a state-of-the-art accuracy for SoLi of greater than 98% is reached, which is superior to any DNN approach. For a non-public radar gesture dataset, in Kreutz et al. (2021) we combine the AoA information with range-Doppler maps from multiple frames to train deep SNNs with surrogate gradients and temporal coding. Different ways of encoding scalar values into spikes are evaluated.

Other SNNs operate on the micro-Doppler patterns: For the IMEC 8GHz dataset, Stuijt et al. (2021) treat the micro-Doppler as a binary image, train a DNN and convert it to a rate-based SNN. For the same dataset, Safa et al. (2021b) improve the classification accuracy by means of time-to-first-spike coding, a direct training of the spiking CNN and further preprocessing. Instead, in Arsalan et al. (2021), we treat the micro-Doppler pattern as a sequence of velocity vectors which is then fed into a SNN consisting of a 1D convolution layer, one dense LIF hidden layer, and an output layer. The network is trained with a SoftLIF (Hunsberger and Eliasmith, 2016) activation (an approximation to LIF) in the NengoDL framework (Rasmussen, 2019). Note that here the conversion to spikes only happens after the first convolutional layer. In a very different approach Banerjee et al. (2020) apply unsupervised learning (STDP) to train the weights of spiking convolution layers on binarized micro-Doppler sequences. A logistic-regression-based classifier acts on the output of all spiking convolutional layers.

In this work, we combine several of the concepts from radar gesture recognition for automotive radar object classification: Sequences of ROIs in the range-Doppler maps are classified with a spiking convolutional network with recurrent layers that was trained using surrogate gradients and BPTT. For details see Section 4.2.

Neurological experiments with different types of mammals have shown, that they do some kind of path integration, i.e., they are able to infer their current position relative to some reference point with the help of e.g., locomotion signals (Etienne and Jeffery, 2004). (Continuous) attractor networks have not only a high degree of biological plausibility but also have been the most successful network type for modeling path integration (Redish and Touretzky, 1997). These networks have already been deployed successfully to real-world problems like mobile robot localization and mapping (Milford et al., 2004), as they are capable of keeping a (Gaussian) state representation continuously, even in the absence of any input.

Since the problem of path integration, which is a subproblem of simultaneous localization and mapping (SLAM), is very similar to the tracking of targets, we expect that this approach can be adapted for tracking objects in range-Doppler or range-angle maps. We further assume that with some changes in the connections and weights of the network it is possible to implement a clustering algorithm, similar to the ones used for clustering the data points in a range-Doppler map.

In such case, both the clustering and tracking could be solved by the same network(cf. Section 3.3.3).

In Section 3.3, we have presented two SNNs that use temporal encoding for replacing the CA-CFAR and OS-CFAR algorithms, respectively. While both of them are mathematically equivalent to the original algorithms, their performance may deteriorate when being realized on neuromorphic hardware due to limited parameter resolution or discretization of spike times.

For SNNs that use temporal coding, especially the binning of spike times to time steps can become a severe constraint: When considering digital neuromorphic systems that have a global system tick for updating neurons or inserting spikes (such as Loihi, TrueNorth, or SpiNNaker), the total number of time steps will have a major impact on the accuracy of the time-coded spiking CFAR networks. To assess this limitation we implement the CFAR SNNs with different number of time steps where the input values are translated to discrete spike times. We compare the output of the spiking CFAR to the reference implementation and provide exemplary results: Figure 6A shows a challenging sample RD map from the CARRADA dataset due to the long extension of the main object and the slow degradation of the intensity until it becomes background. Figures 6B,C show the performance of the spiking OS- and CA-CFAR, respectively, when simulated for 250 time steps. We then count the number of true positives (TP: targets detected by both classical and spiking algorithms), false positives (FP: targets detected by spiking CFAR but not by conventional), and false negatives (FN: true targets not identified by spiking CFAR). The examples show many true positives, as well as several false negatives and few false positives. The detected bins by the classical algorithms differ slightly between OS and CA-CFAR due to the different approaches for noise level estimation. It further stands out that the spiking OS-CFAR has more false negatives than the spiking CA-CFAR but no false positive detections. Details on the chosen CFAR parameters are given in Supplementary Section 1.1.

For a statistical analysis we evaluated the spiking CFAR on 1,000 randomly selected RD maps from the CARRADA dataset and accumulated the counts of TP, FN and FP detections. Based on this we obtain the sensitivity and precision as performance indicators of the spiking variants: (13)sensitivity=TPTP+FN, (14)precision=TPTP+FP. For both metrics a value close or equal to 1 is desired. The metrics are evaluated depending on the number of SNN simulation time steps in Figure 7: the spiking CA-CFAR with nearest-rounding for binning spike times to simulation time steps shows a very high sensitivity even for less than 100 time steps, which means that only few CFAR detections are missed by the SNN. However, there are many false positive detections, so that the precision stays below 95% for less than 300 time steps. Only starting from 500 time steps both sensitivity and precision are above 99%, which we consider competitive to the original algorithm. We tried additional rounding schemes for the CA-CFAR, which are introduced and discussed in Supplementary Section 1.2.2, exhibiting a worse performance than the rounding to nearest presented here.

In contrast, the results of the spiking OS-CFAR in Figure 7B show a different dependency: There are no false positives at all (precision is always 1) while the sensitivity increases very slowly with the number of time steps and reaches 95% at around 800 time steps. The reason for this bad performance of the spiking OS-CFAR can be explained by looking at the distribution of input values: When using RD map amplitudes as inputs, most of the converted spike times are binned to only a few number of time steps which leads to missed detections by the SNN (see Supplementary Section 1.3.1 for details). Alternative rounding schemes are not considered here, as they do not affect the order statistic. Instead, we converted the RD map to a logarithmic scale prior to feeding the values to the network. Moreover, we added a small time delay to the neighbor cells in order to avoid false negatives when the center cell is slightly bigger than the k-th largest value. These modifications increased the sensitivity of the spiking network to around 99% when simulating it for 100 time steps (see Figure 7C). Further details and explanations are provided in Supplementary Section 1.3.

To sum up, we found that the performance of the algorithms can reach values close to 99% when using an adequate amount of time steps. The required time steps are lower for the spiking OS-CFAR, mostly thanks to the logarithmic re-scaling of the input range-Doppler map (which is not possible for the CA-CFAR). We note that, when the spiking CFAR algorithms are embedded into a full radar processing chain with subsequent classification and tracking, a lower performance with respect to the classical CFAR might be sufficient: One could co-optimize the parameters of the spiking CFAR and the classification algorithm to achieve a high overall performance, e.g., one could decrease the CFAR threshold factor to create more detections and let the classifier filter out the noise from actual radar targets.

Finally, we compare the computational effort of the spiking CFAR algorithms to the conventional approaches. For SNNs the effort mainly depends on the number of neuron updates and synaptic events. Both spiking CFAR networks perform as many neuron updates as time steps N_steps and process as many synaptic events as training cells N_train. Additional effort is required for releasing input spikes at the predefined times, which, however, only needs to be done once per RD map bin if the CFAR is realized by one large SNN for the whole RD map. The classical CA-CFAR is dominated by N_train ADD operations. The OS-CFAR, which compares the kth largest element with the cell under test, can be efficiently implemented by N_train compare operations (a sorting of training cells is not required).

Table 1 evaluates the computational cost in terms of ADD and compare operations: The spiking OS-CFAR requires approximately N_steps more operations than the classic approach, and the spiking CA-CFAR needs 2N_steps more operations. Considering that in our example there are 176 training cells and 100 time steps (OS-CFAR) resp. 500 time steps (CA-CFAR), it is apparently not beneficial to realize the CFAR algorithms as spiking networks on conventional processors. Yet, a spiking CFAR network might be realized very efficiently on dedicated neuromorphic hardware, especially when the input is already provided as spikes, or when the spiking output is directly fed into subsequent SNNs. For the future, we suggest to directly compare the energy and latency of the spiking CFAR on neuromorphic hardware to the conventional CFAR on a suitable DSP.

To evaluate the feasibility of spiking networks for object classification based on range-Doppler maps, the CARRADA dataset is used. As the conceived network is only for the purpose of object classification (and not object detection or localization), we prepare a sub-dataset that only contains fixed sized regions of interest for all labeled objects in the range-Doppler maps of the dataset. To include temporal dependencies the extracted regions are taken as small sequences of 8 frames. The generated sub-dataset contains 399 car, 208 bicycle, and 323 pedestrian sequences. Additionally, we consider the dataset of all single ROIs for training a 2D convolutional network as a baseline reference. The dataset preparation is elaborated in the Supplementary Section 2.1. We note that in a real-time scenario the ROIs need to be selected dynamically based on the location of detected objects.

Figure 8 visualizes the proposed network architecture for the classification of ROI sequences. Two variants of the neural network are considered: First, an ANN consisting of two convolutional layers, a recurrent layer with LSTM cells, and the output layer. Second, an SNN with two spiking convolutional layers, a recurrent layer with LIF neurons, and an output layer with non-spiking integrator neurons. Both networks have the same structure and layer sizes which are detailed in Supplementary Section 2.3. In both cases, the 2D convolutional layers extract spatial information from single frames while the recurrent layer combines the latter for spatio-temporal signal processing. The proposed SNN model resembles the spiking convolutional network for gesture classification from Safa et al. (2021b); yet it was developed independently and differs from it by having recurrent connections between the LIF neurons. Both the ANN and SNN in this work are trained on real-valued inputs and on event-encoded inputs. This allows to analyse the effect of converting the input data to spikes on the overall network performance. To generate the input spikes, the range-Doppler map values are compared to a specific reference value. With this scheme, on average 537 input spikes are used to encode the sequential ROIs including 8 discrete time steps. In case of the ANN the encoding is used to create a binary input map. In contrast, the first convolutional layer of the spiking network is also evaluated using real-numbered inputs in combination with a spiking activation function similar to some converted SNNs (Hunsberger and Eliasmith, 2016; Rueckauer et al., 2017). The network models are trained with BPTT, using surrogate gradients in case of the SNN. In addition to the recurrent networks, a 2D CNN is trained on single frames providing a baseline reference. Further details on the network architectures and training methods can be found in Supplementary Section 2.

Table 2 shows the results of the conducted experiments: The best accuracy of 94.7% is achieved by the ANN with convolutional and LSTM layers applied on ROI sequences, significantly better than the CNN on single frames (90.5%) and the pure SNN with spike input (90%). When the recurrent ANN is trained with binary inputs, the accuracy drops down to 86.3%. In contrast, when the SNN processes the first convolutional layer with real-valued inputs, the accuracy is increased to 92.6% getting closer to the ANN. The table also provides the number of parameters and the number of operations [spikes, synaptic events, and multiply-accumulate (MAC) operations] per network model as a measure of the network complexity and computational cost. The SNN with spike input clearly shows the best compromise between number of operations and achieved accuracy. Also, the SNN with real-numbered inputs still requires less than 20% of operations of the best performing ANN.

Note that, in contrast to the spiking CFAR evaluation, we do not consider specific limitations of neuromorphic hardware like the number of time steps, which is 8 in this experiment for all recurrent networks. Yet, the results show that the binary spike encoding decreases the accuracy for both the ANN and SNN. We further remark that the dataset is rather small and we expect that the test accuracy can be improved for all models by increasing the training data and a thorough hyperparameter optimization. Nonetheless, this example demonstrates the general feasibility of SNNs for efficient object classification with automotive radars.

In this article, we reviewed the state-of-the-art digital signal processing steps for automotive radars and discussed for each step various SNN approaches as replacements (Section 3). To the best of our knowledge, such comprehensive analysis of concepts for radar processing with SNNs has not been done before. Yet, we consider this collection of approaches preliminary, and we are sure that more and enhanced approaches will be adopted or developed in the future. Furthermore, for two processing steps we have provided concrete SNN examples and compared them to classical approaches: For the CFAR object detection we developed two temporally coded SNNs and analyzed their accuracy depending on time steps. Starting from 100 time steps, the spiking version is competitive with the reference approach. For object classification, we trained a deep recurrent SNN with BPTT and surrogate gradients on ROI sequences of range-Doppler maps from the CARRADA dataset. The accuracy of the SNN with real-valued inputs of 92.6% is close to the 94.7% achieved by a reference ANN while requiring only 18% of the operations. Instead, the pure SNN with spike input achieves 90.0% with less than 0.5% of the operation of the ANN. Further improvement is expected by increasing the size of the dataset and performing a systematic hyperparameter search. Regarding related work in the context of FMCW radar, so far, SNNs have only been used for gesture recognition, cf. Section 3.4. Very recently, Stuijt et al. (2021) have demonstrated radar gesture recognition using an ultra-low-power SNN chip and a 8 GHz FMCW radar. They turn the micro-Doppler map into a small binary image and classify it with a rate-based feed-forward SNN on the chip. In López-Randulfe et al. (2022), the time-coded spiking Fourier transform introduced in Section 3.1.2 was implemented and validated on Loihi to compute the range and Doppler-FFT on recorded radar data. Compared to dedicated hardware FFT accelerators, the neuromorphic solution lags behind by one to three orders of magnitudes in terms of energy and latency. Brown et al. (2021) have developed an SNN hardware accelerator for compressed sensing with pulse-Doppler radars. A spiking locally competitive algorithm (LCA) solves the sparse optimization to achieve highly accurate and efficient target and velocity estimation. This compressed sensing approach is not directly applicable to the FMCW automotive radar processing chain discussed in this article. Further, Barnell et al. (2020) use spiking DNNs on Loihi for classification of synthetic aperture radar images. While this demonstrates the efficiency of neuromorphic hardware for image classification, new network models will have to be developed for automotive FMCW radar data.

The SNN concepts presented in this work apply to single steps of the radar processing chain. How to combine several SNNs or how to build a radar processing chain completely with spiking neurons was not the objective of this paper and remains an open research subject.

Whether or not spiking neural networks can outperform conventional radar processing depends on how efficiently they can be realized in neuromorphic hardware. In the following, we summarize the requirements of a neuromorphic application-specific integrated circuit (ASIC) to process the radar data in real time. For this, we assume that a neuromorphic processor replaces or complements a DSP (cf. Figure 2A) and receives raw ADC data or preprocessed data that has to be converted to spikes on the chip. Our analysis includes the required memory for buffering input data, the required input bandwidth, the number of neurons and synapses, and the processing speed of those neuromorphic components. As a radar sensor setup we take the one from the CARRADA dataset with 2 transmitters and 4 receivers (cf. Table 3), yet we note that the requirements for high-resolution radars will strongly increase. Reviewing the radar processing steps from Figure 3, the hardware requirements vary significantly for each processing step, e.g., the amount of input data per frame that needs to be processed varies a lot, as shown in Table 3. Especially processing the full raw data or high-resolution range-angle maps requires more than 100 kB of memory for buffering the input. This amount does not pose a problem for typical embedded micro-processors, yet it might become challenging for high-resolution radars with more than 10 times as much data or when fed into edge neuromorphic processors. Similarly, for the communication between a radar sensor and neuromophic hardware at least a bandwidth of 10–100 MBit/s is needed.

At the bottom of Table 3, we review SNN requirements for some of the radar processing steps: The range S-FT with time coding from López-Randulfe et al. (2022) can be realized with sparse connectivity and one spike per synaptic connection for 550 time steps. While the S-FT network itself is rather small, the challenge is to run the model 256 times (64 chirps × 4 receivers) per frame on a neuromorphic processor (e.g., within 20 ms assuming that 20% of the 100 ms frame time are budgeted for the range-FFT). This seems possible, according to the results obtained in López-Randulfe et al. (2022), where a 1024-point spiking FFT can be calculated every 105 μs on the Loihi neuromorphic chip. For the spiking OS-CFAR from Section 4.1, a network of 16 k input and output neurons with nearly 3 million synapses is required to process an entire range-Doppler map. Compared to the range-FFT, this SNN is run only once per frame and thus has lower neuromorphic compute demands. Finally, the SNN-based radar object classification (Section 4.2) has the least requirements for implementation on neuromorphic hardware as the network is smaller and there is only one time step per frame (cf. Supplementary Section 2.3). Note, however, that the network needs to be newly instantiated for each detected object and we expect in the order of up to 20 radar objects in simple street scenes.

Looking at the neuromorphic requirements for the different automotive radar processing steps, we expect that SNN-based object classification has the highest potential for energy-efficient realization in neuromorphic hardware. SNN-based object tracking should also be evaluated further in the future. For the earlier processing steps like the FT and CFAR object detection, further work shall determine if neuromorphic hardware tailored at these operations can implement these operations more efficiently than digital signal processors and close the current gap in terms of energy and time performance (López-Randulfe et al., 2022). At the system level, one could alternatively combine a DSP with a neuromorphic processor to achieve maximum efficiency. When split onto different chips, the data bandwidth requirements from Table 3 need to be fulfilled. An even more radical approach for radar processing with neuromorphic hardware is to use analog spiking neurons in hardware with the radar IF signal as input. Resonate-and-fire neurons are the perfect candidates for that, but this might be limited to radar systems that don't need phase information, e.g., using a single transmitter and receiver. Yet, further research may clarify whether a full SNN pipeline on dedicated neuromorphic hardware can outperform classical DSP or hybrid DSP/ANN approaches.

As motivated in the introduction, the use of neuromorphic hardware has a high potential to significantly reduce the energy demands for highly-automated driving. Besides radar signals, also camera and LIDAR data need to be processed in order to get a complete understanding of the automotive scene. For image processing there already exist first attempts to solve complex tasks with SNNs, e.g., for object detection (Kim et al., 2020) or semantic segmentation (Kim et al., 2021). Also recently, Viale et al. (2021) realized an SNN on Loihi for car detection using a dynamic vision sensor. Using LIDAR data, which is naturally sparse and thus predestined for SNNs, Zhou et al. (2020) showed a spiking convolutional network for real-time 3D object detection. Shalumov et al. (2021) use LIDAR data for SNN-based collision avoidance with a control network based on the neural engineering framework. All these examples show that SNN-based sensor processing for autonomous driving is a trending topic. Besides the development of SNNs and their implementation on neuromorphic hardware, also the combined processing, i.e., sensor fusion using SNNs, will become an important topic.

When it comes to AI-based autonomous driving, ensuring functional safety of both software and hardware is a critical issue. The principles that are currently developed to support machine learning models (Henriksson et al., 2018; Mohseni et al., 2019) will also apply to SNNs. Similarly, neuromorphic hardware will have to fulfill the same standards as any automotive electronic system: adhere to temperature ranges, be resistant to vibrations, be deterministic and redundant, or contain self-monitoring. For that reason, only digital neuromorphic systems are candidates for integration in cars, while the use of analog or mixed-signal neuromorphic hardware seems out of scope at the moment due to their intrinsic variability. Hence, we suggest to focus on advanced digital systems such as SpiNNaker2 (Yan et al., 2021) or Loihi2 (Orchard et al., 2021) to further explore neuromorphic hardware for automotive radar processing and automated driving in general.