The brain's circuits are massive in terms of numbers of neurons and synapses, suggesting that large circuits are fundamental to the brain's computing. Hyperdimensional computing (HDC) (Kanerva, 2009) explores this idea by looking at computing with ultra-wide words—i.e., with very high-dimensional vectors or hypervectors. The fundamental computation units in HDC are high dimensional representations of data known as “hypervectors” constructed from raw signals using an encoding procedure. There exist a huge number of different, nearly orthogonal hypervectors with the dimensionality in the thousands (Kanerva, 1998). This lets us combine such hypervectors into a new hypervector using well-defined vector space operations while keeping the information of the two with high probability. Hypervectors are holographic, that is, the information encoded into the hypervector is distributed “equally” over all the components. In our case, it is done using (pseudo)random hypervectors with i.i.d. components as our ingredients for the encoding. A hypervector contains all the information combined and spread across all its components in a full holistic representation so that no component is more responsible for storing any piece of information than another.

In HDC, hypervectors are compositional—they enable computation in superposition, unlike standard neural representations (Kanerva, 2009). These HDC operations allow us to reason about and search through images that satisfy pre-specified constraints. These composite representations can be combined using HDC operations to encode temporal information or complex hierarchical relationships. This capability is especially powerful for understanding the relationship between objects in images in both time and space. These operations are simple in HDC and require only trivial element-wise arithmetic. By contrast, to achieve the same effect in a neural network, e.g., spiking neural networks (Wang et al., 2018), we would need to assign images corresponding to composite classes a new label and train a separate model for prediction. HDC also provides a natural way to preserve temporal information using a permutation operator (Rahimi et al., 2016b). For example, we encode a sequence of video frames while preserving the temporal structure. This would allow us to efficiently compute a similarity score for entire sequences of video using a standard HDC similarity search, which is extremely efficient in hardware (Li et al., 2016).

Let us assume H→1, H→2 are two randomly generated hypervectors (H→∈{-1,+1}D) and δ(H→1,H→2)≃0, where δ is the cosine similarity function, δ(H→1,H→2)=H→2·H→2∥H→1∥·∥H→2∥.

Binding (*) of two hypervectors H→1 and H→2 is done by component-wise multiplication (XOR in binary) and denoted as H→1*H→2. The result of the operation is a new hypervector that is dissimilar to its constituent vectors i.e., δ(H→1*H→2,H→1)≈0; thus, binding is well suited for associating two hypervectors. Binding is used for variable-value association and, more generally, for mapping.

Bundling (+) operation is done via component-wise addition of hypervectors, denoted as H→1+H→2. The bundling is a memorization function that keeps the information of input data in a bundled vector. The bundled hypervectors preserve similarity to their component hypervectors i.e., δ(H→1+H→2,H→1)>>0. Hence, a bundling of hypervectors is well suited for representing the set of elements corresponding to the hypervectors that are bundled, and we may test their membership by a similarity check.

Permutation (ρ) operation, ρn(H→), shuffles components of H→ with n-bit(s) rotation. The intriguing property of the permutation is that it creates a near-orthogonal and reversible hypervector to H→, i.e., δ(ρn(H→),H→)≃0 when n ≠ 0 and ρ-n(ρn(H→))=H→. Thus, we can use it to represent sequences and orders.

Reasoning is done by measuring the similarity of hypervectors. We design the encoding of the hypervectors such that the similarity between the hypervectors reflects the similarity between the entities that they represent.

This article focuses on learning over data collected by the Dynamic Vision Sensor (DVS). Unlike a normal camera that captures data synchronously and frame-based, a DVS camera mimics the mechanics of the human retina by detecting and recording the changes in the illumination of a pixel asynchronously, sending a stream of events to the memory. This leads to sparse data because only a small subset of pixels reports events at any time, with rich temporal information, because of the asynchrony, rendering it much more difficult to train. Nevertheless, DVS data has been actively studied and researched in the context of neuromorphic computing, e.g., in conjunction with the Spiking Neural Network, for various image-related tasks such as gesture recognition and object classification (Massa et al., 2020).

In this article, we present EventHD, an end-to-end framework for robust, efficient hyperdimensional learning from the neuromorphic sensor. Unlike all prior works that operate over preprocessed data, to the best of our knowledge, EventHD is the first HDC-based solution that directly operates over raw neuromorphic data. We first develop a novel hyperdimensional encoding scheme to map event-based neuromorphic data into high-dimensional space. EventHD exploits hyperdimensional mathematics to preserve spatial and temporal information from raw sensor data (Section 3). Next, we introduce novel algorithm solutions to perform classification and self-learning over the encoded data (Section 4). This includes enabling single-pass classification and supporting association and memorization over perception-action space (Section 5).

Hyperdimensional computing encoding is performed based on a set of base or seed hypervectors. The base hypervectors represent the basic alphabet of the data. For an example of DVS data, the alphabets are illumination changes and the position of events. Figures 1A,B shows the EventHD base generation procedure.

In a given time, our encoder looks at neuromorphic data as an image with few activated spikes/events. The goal of the encoder is to map this data into high-dimensional space using pre-generated base hypervectors. The encoding is performed in two steps, as shown in (Figure 1C):

Associating event-illumination: For every activated event, our encoder exploits a binding operation to associate each event position with the corresponding illumination hypervector. For example, if an event in position [i, j] is activated, our encoder associates information using: P→i,j×L→i,j, where L→i,j can be L→+1 or L→-1 depending on illumination direction. The bound hypervector preserves position and illumination information in a new hypervector that is nearly orthogonal to its operands. We perform the same association for all activated events.

Event memorization: In HDC, bundling acts as memorization. We exploit this feature to memorize the information of all activated events in a given time. Our solution bundles associated hypervectors for all activated events: S→=∑i=1r∑j=1c(P→i,j*L→i,j) when (r, c) has a spike event. The memorization and summation only happens for pixels that have spike events.

Let us consider actual neuromorphic data with temporal spikes/events. As we explained in Section 3.2, for all events that happen in a time window, we exploit spatial encoding to map all events into single hypervectors. As time moves on, the information of new events needs to be encoded into a new hypervector. Figure 2 shows two solutions to memorizing signals and keeping temporal information.

Permutation-based: To incorporate a notion of time, our encoder represents the position of each time slot using a single permutation. The permutation in HDC is defined as a rotational shift, where the permuted hypervector is nearly orthogonal to its original vector. Figure 2A shows how n spatial-encoded data can be temporally combined through time. For example, to memorize three consecutive encoded signals, S→1, S→2, and S→3, we encode them into a single hypervector by H→=ρ1(S→1)*ρ2(S→2)*ρ3(S→3), where binding (*) and permutation (ρ) memorize sensor value and position. This encoding preserves the temporal information of events. Although permutation can preserve sequence information, it is a very exclusive operation that loses the information of continuous-time. For example, even when two continuous events are identical, this temporal encoding is orthogonal (δ(ρ1S→,ρ2S→)≃0).

Association-based: To give a notion of continuous time, we exploit binding operations to keep temporal correlative information. Our temporal encoding is performed using the following steps: (1) Similar to spatial encoding, we generate a set of correlated hypervectors to preserve temporal correlation. As Figure 2B shows, our solution splits time into smaller t-size windows. We generate a random hypervector for each time that is a factor of t. For example, we generate random hypervectors representing {T→0,T→t,T→2t,⋯,T→kt}, where indices represent time steps. (2) We perform interpolation to generate a correlated hypervector representing intermediate times. Given time t₀ ∈ [jt, (j + 1)t] for some non-negative integer j, T_t0 is generated by taking components from T_jt and T_(j+1)t in (j + 1)t − t₀:t₀− jt ratio such that the similarity between the three reflects their original correlation. For an example of t = 3, T→1 will be 66.6% similar to T→0 and 33.3% similar to T→3. Our temporal correlation goes beyond a single-window; hypervectors in two neighbor windows are also correlated.

As Figure 2 shows, we exploit the time-base correlated hypervectors to preserve temporal information. Let us assume H→i is a hypervector of events happening in a time slot i. Our encoding preserves temporal correlation of p time-slot using: H=∑i=1p(T→i*S→i).

EventHD supports two types of classification: accumulative and adaptive learning. Both methods are a single-pass approaches that can construct a learning model by one-time looking at training data. The single-pass model is significantly fast and efficient and enables learning from the data stream with no need for off-chip memory.

Accumulative training (single-class update): Hyperdimensional computing models receive their dataset as copies of the memory component at the point of evaluation. To find the universal property for each class in the training dataset, the trainer module linearly combines hypervectors belonging to each class, i.e., adding the hypervectors to create a single hypervector for each class. Once combining all hypervectors, we treat per-class accumulated hypervectors, called class hypervectors, as the learned model. Figure 3 shows HDC functionality during single-pass training. Assuming a problem with k classes, the model represents using: M={C1→,C2→,⋯,Ck→}. For example, after generating all encoding hypervector of inputs belonging to class/label l, the class hypervector Cl→ can be updated using: Cl→=∑jJ(1-δ(H→j,C→l))×H→j, where there are J inputs having label l. This weighted data accumulation continues for all train data available in each class. Accumulative training gives a rough estimation of a pattern of each class hypervector. However, it does not find a chance to adjust the class hypervectors for marginal predictions. This makes the HDC model sensitive to possible noise in the input data.

Adaptive training (multi-class update): We propose adaptive training that not only accumulates each train data with the correct class but also updates the class hypervectors with a possible marginal match. EventHD checks the similarity of each encoded query data with all class hypervectors. If an encoded query H→ corresponding to label l, the model miss-predicts it as label l′, the model updates 2 × i neighbor classes using the following equation:

where δl=δ(H→,Cl→) and δl′=δ(H→,Cl′→) are the similarity of data with correct and miss-predicted classes, respectively. Unlike the accumulative training, our adaptive update provides two main features: (i) it updates multiple class hypervectors, which are centered around correct and miss-predicted classes. The neighbor class hypervector gets updated depending on its physical distance to the query (η_i sets the update ratio). This method ensures that class hypervectors have a smoother pattern of similarity; thus, a small noise in the input data cannot cause miss-prediction. (ii) Adaptive training also ensures that we update the model adaptively based on how far a train data point is miss-classified with the current model. In case of a far miss-prediction, δl′>>δl, retraining makes major changes to the mode. While for marginal miss-prediction, δl′≃δl, the update makes smaller changes to the model.

Inference: checks the similarity of each encoded test data with the class hypervector in two steps. The first step encodes the input to produce a query hypervector H→. Then, as Figure 3 shows, we compute the similarity (δ) of H→ and all class hypervectors. Query data gets the label of the class with the highest similarity.

EventHD also supports online self-learning where only a small portion of training data is labeled. EventHD exploits the HDC model transparency to improve the quality of the model. Using the techniques introduced in Imani et al. (2019a), it checks the similarity of each unlabeled data with the already trained model, obtaining the confidence level of the classification result. If the confidence level is higher than a threshold (e.g., α > 90%), EventHD updates the model by embedding encoded data into the corresponding class hypervector, as: C→max=C→max+α×H→, where H→ is the query data and C→max is a class that has the maximum similarity with a query. EventHD exploits this same technique to update the model based on the user's feedback on the inference results. Given the absence of labels in the majority of observations, we assume that users would be willing to provide feedback when they are not satisfied and tune the confidence threshold accordingly.

Hyperdimensional computing can naturally correlate them in high-dimensional space (Mitrokhin et al., 2019). This association enables EventHD to reason about each prediction by giving systems prior knowledge. Let us consider a system with n-feature as perception (x→={f1,f2,⋯,fn}) and m-output actions (y→={a1,a2,⋯,am}) in original space. Our approach encodes both perception and action data into high-dimensional space. For perception, we exploit the proposed encoding, explained in Section 3, that preserves the spatial correlation of events. However, the output actions are often independent and do not have any spatial correlations. Therefore, our encoding method randomly generates the position hypervectors, rather than generating correlated position hypervector for a given image data. X→=∑k=1mP→k*L→∈F, where δ(P→i,P→i+1)~0.

EventHD also encodes the output action into high-dimensional space. The action is often a single output signal. Our method linearly or non-linearly quantizes the action signal and assigns a hypervector to each quantization level, {A→1,A→2,⋯,A→m}. Our solution naturally associates each pair of perception and action by binding their corresponding hypervector. The accumulation of the bound vectors over prior observations gives native HDC-based memorization to the system: S→=∑i=1nX→i*A→i. Let us assume each reference hypervector store N encoded perception-action hypervector: R→=S→1+⋯+S→N=∑j=1NS→j. We can predict an action for a perception X→k, using:

where A→k is an interpolation between all actions that their perceptions have high similarity to X→k. Figure 4A shows EventHD selecting between two discrete actions. Depending on the confidence, i.e., the similarity of a query to memorized perceptions, EventHD picks one of the actions. In continuous space, the selection translates to interpolation between the actions, depending on the perceptions similarity in HDC space.

In HDC, bundling acts as a memory, storing the information of multiple encoded hypervectors into a single reference hypervector. EventHD exploits bundling to memorize the associated perception-action, R→=∑j=1NS→j. The reference hypervector has limited capacity and, thus, cannot store the information of unlimited encoded data. The capacity depends on the dimensionality and the orthogonality of the encoded hypervectors. For a given query data, EventHD can refer to memory in order to retrieve the system's prior knowledge. For example, let us assume q is a perception with Q→ being its encoded data. EventHD can retrieve information about possible actions by checking the similarity of the query with the reference model:

If P→λ=Q→ for some λ, the output of the function is going to be A→λ. For reference patterns that do not match with the query, the similarity is nearly zero, δ(S→i,Q→)≃0. Thus, we can check the existence of a query Q→ in R→ using the following criteria: δ(R→,Q→)/D>T, where T is a threshold and δ(R→,Q→)/D is called the decision score.

Figures 4B,C show the normalized distribution of signal and noise in EventHD information retrieval (using D = 10k). These Gaussian distributions determine the capacity of each reference hypervector in memorizing the information. As our mathematical model indicated, the noise is getting a wider distribution when increasing the number of patterns stored in R→. When the noise overlaps with the signal, there is no threshold T that can separate noise, thus resulting in information loss (Figure 4C). Figure 4D shows the capacity of reference hypervector with D dimensionality in storing N nearly orthogonal and correlative patterns. Our evaluation shows that the capacity of the reference hypervector increases with dimensionality. For example, EventHD with D = 4k can stored N = 10³ (N = 10⁴) orthogonal patterns with less than 0.5% (10%) information loss. Note that, in practice, the reference hypervector has a much higher capacity as EventHD encoder keeps the correlation between input signals. As Figure 4D shows, a reference hypervector provides significantly higher capacity when the EventHD encoder preserves correlation in high-dimensional space. For a more in-depth analysis of the memory capacity, readers are referred to Frady et al. (2018).

EventHD similarity search on the memorized model gives us an estimation of the output action. EventHD uses this prediction as prior knowledge to trust the prediction. If the prediction is relatively far from the memorized action, EventHD gives very low confidence to that prediction. This approach enables us to reason about each prediction and potentially provide a more explainable learning solution.

We implement EventHD using software, hardware, and system implementation. In software, we verified EventHD training and testing using a C++ implementation. For hardware, we design the EventHD functionality using Verilog and synthesize it using Xilinx Vivado Design Suite (Feist, 2012). The synthesis code has been implemented on the Kintex-7 FPGA KC705 Evaluation Kit. We ensure our efficiency is higher than the automated FPGA implementation at (Salamat et al., 2019).

We evaluate EventHD accuracy and efficiency on two Datasets: the Neuromorphic MNIST (N-MNIST) and the Multi-Vehicle Stereo Event Camera (MVSEC) dataset. N-MNIST is an event-based version of the MNIST dataset, containing event-stream recordings of the 60,000 training digits and 10,000 testing digits. The MVSEC dataset collects event-based DVS cameras on the self-driving car day and night (Zhu et al., 2018; Mitrokhin et al., 2019). This dataset is designed for regression tasks to predict the car velocity based on DVS data. The experiments correspond to mDAVIS-346B cameras with 346×260 pixel resolution. To find ground truth velocity values, the car is equipped with IMUs and GPS sensors. The evaluation is performed for five activities, two recorded during the day and three in the evening/night. The results are reported using two metrics: Average Relative Pose Error (ARPE) and Average End-point Error (AEE). ARPE shows the average angular error between translational vectors while ignoring the scale (Mitrokhin et al., 2019), while AEE shows the absolute error in 2D linear space. Similar to other error metrics, the lower ARPE and AEEE indicate higher quality of learning. All results are reported for MVSEC data unless they are stated differently. We use a simple DNN with one 512-neuron hidden layer as our baseline from conventional neural networks. EventHD is configured to have a hyperdimension of D = 4, 000, a window size of k = 5 for positional, and a time window size of t = 50(ms) across all experiments, as it leads to the best average performance.

Figure 5A compares EventHD quality of learning over classification task, using both day and night data. We compare EventHD with state-of-the-art HDC methods working on event-based sensors: DNN, Dense HDC (DenseHD) (Mitrokhin et al., 2019), and Sparse HDC (SparseHD) (Hersche et al., 2020). All three baseline approaches operate over six extracted features by the preprocessing method. In contrast, EventHD is an end-to-end framework that directly operates over raw neuromorphic data. Note that other algorithms, i.e., DNN, DenseHD, and SparseHD, provide close to a random prediction when processing the raw neuromorphic data. For EventHD, we report the results for single-class (Single-C) and multi-class (Mult-C) updates using both ARPE and AEE metrics. Our evaluation shows that EventHD using both accuracy metrics provides comparable or better quality of learning compared to the state-of-the-art solutions. For example, EventHD ARPE (AEE) error metric is, on average, 0.1% and 4.8% (37.0% and 14.1%) lower than DenseHD and SparseHD, respectively. These metrics indicate EventHD higher quality of learning. Note that EventHD efficiency and robustness are significantly higher than all baseline methods due to eliminating costly preprocessing (detailed evaluation in Section 6.3).

Figure 5B also evaluates the EventHD quality of learning on the N-MNIST dataset. The results are compared to SNN and HDC-based neuromorphic approaches. Unlike EventHD which operates over raw neuromorphic data, SparseHD and DenseHD rely on preprocessing algorithms to extract spatial-temporal information. Our evaluation shows that EventHD provides significantly higher classification accuracy than existing HDC-based algorithms, i.e., SparseHD and DenseHD.

Temporal encoding: Figure 5 also compares EventHD accuracy using permutation-based (ρ) and association-based (*) temporal encoding. Our evaluation shows that the association-based encoding provides a lower error rate by enabling a notion of continuous-time dynamic, while permutation-based encoding only preserves the orders of events. For example, EventHD using association-based provides 10.2% (17.2%) lower ARPE (AEE) compared to a permutation-based solution on the MVSEC dataset.

Single vs. multi-class: Figure 6 visually compares EventHD classification accuracy in two configurations over the MVSEC dataset: a single-class and a multi-class update. In both configurations, we show the final prediction (Figure 6A) and the similarity of a query with different class hypervectors (Figure 6B). EventHD with a single-class update creates a weak learning model with high sensitivity to noise and variation in the input data. Therefore, during inference, it may deviate toward the wrong class. However, our multi-class update solution keeps the correlation between the predicted speeds and strength of the class hypervectors, thus providing higher learning accuracy. The box in Figure 6 clearly shows the capability of EventHD multi-class update to strengthen the signal in related class hypervectors and provide a higher quality of prediction.

Robustness to variation: Unlike prior HDC-based approaches that do not keep the correlation, EventHD encoding is asynchronous, thus preserving both temporal and spatial correlation over event-based data. We perform an experiment to show EventHD capability to respond to noisy data. Figure 7A shows EventHD and HDC quality of learning when the activated events in each timestamp are randomly shifted in an arbitrary direction. Our evaluation shows that EventHD is highly robust against such possible variational data, as it provides the maximum accuracy even using a 5% shift. In contrast, the state-of-the-art HDC solutions do not keep the correlation between neighbor pixels (spatial correlation). Therefore, a single shift operation can generate a signal which is entirely orthogonal to the non-shifted version. As Figure 7A shows, this makes the existing HDC solutions, DenseHD and SparseHD, very sensitive to possible noise or variation in the input signal.

Self-learning: Figure 7B shows EventHD classification accuracy during the self-learning iterations. The results are reported when EventHD has been trained, supervised over 10% of train data, and unsupervised over the other 90%. Our evaluation shows that EventHD can adaptively improve classification accuracy during self-learning. This advantage comes from the EventHD capability of computing confidence for each prediction. Therefore, EventHD trusts data with high confidence for model updates while ignoring low confidential data. On another side, a higher confidence threshold increases the required train samples to converge to maximum accuracy.

We compare EventHD efficiency and robustness to state-of-the-art HDC solutions. The results are for the total processing time, including both preprocessing and learning. The existing HDC solutions use image-to-time transformation as a preprocessing step for feature extraction from the event-based information. The feature extraction result is only a few feature data (i.e., six features in our example). The preprocessing makes the learning task very simple such that even a simple learning solution, e.g., linear regression or perceptron, can provide acceptable accuracy. Due to the complexity of the preprocessing step, its cost eliminates the effectiveness of HDC in enhancing system efficiency. In contrast, EventHD is an end-to-end solution, directly operating over the raw data received by the event-based camera. Our solution eliminates the costly preprocessing step by enabling HDC encoding to preserve both the temporal and spatial locality of the raw data. This improves not only EventHD computation efficiency but also provides significant computational robustness.

Efficiency: Figure 8 compares EventHD computation efficiency with the existing HDC solutions running on FPGA. The results are reported for both training and inference phases. For DNN, we used DNNWeaver V2.0 (Sharma et al., 2016) for the inference and FPDeep (Geng et al., 2018) for training implementation on a single FPGA device. For DenseHD and SparseHD, we use the F5-HD (Salamat et al., 2019) framework for FPGA implementation. All FPGA implementations are optimized to maximize performance by utilizing FPGA resources. All results, in Figure 8, are relative to DNN performance and energy efficiency. During training, EventHD achieves, on average, 10.6× faster and 16.3× more energy-efficient computation as compared to FPGA-based DNN implementation, respectively. The high efficiency of EventHD in training comes from EventHD capability in (i) creating an initial model that significantly lowers the number of required retraining iterations and (ii) eliminating the costly gradient for the model update. This results in higher EventHD efficiency, even in terms of a single training iteration. In inference, EventHD provides 4.3× faster and 6.8× higher energy efficiency as compared to FPGA-based DNN implementation. As compared to SparseHD (DNN), EventHD provides 1.9× and 2.1× (14.2× and 19.8×) faster and more energy-efficient training. The main computation efficiency comes from eliminating the costly preprocessing step and replacing it with HDC encoding.

Robustness: The noise in today's technology is coming from multiple sources. Unfortunately, the existing data representation has very low robustness to noise in hardware. An error bit on the exponents or Most Significant Bits (MSBs) result in a major change in the weight value, while an error in the Least Significant Bits (LSBs) adds minor changes to the computation. The randomness of the noise makes traditional data representations vulnerable to an error on the hardware. One of the main advantages of EventHD is its high robustness to noise and failure. In EventHD, hypervectors are random and holographic with i.i.d. components. Each hypervector stores the information across all its components so that no component is more responsible for storing any piece of information than another. This makes a hypervector robust against errors in its components. EventHD robustness depends on the hypervector dimensionality that determines the hypervector capacity and redundancy. Table 1 compares EventHD robustness with the existing HDC and learning solutions operating the preprocessing or the entire learning task over the original data representation. The results indicate that EventHD quality of learning is almost constant, even using 5% noise. In contrast, even a small amount of error on an existing solution can result in significant quality loss. For example, under 20% random noise, EventHD using D = 4k provides 17.8% and 14.7% higher accuracy than DenseHD and SparseHD, respectively. Note that EventHD robustness increases with its dimensionality (as shown in Table 1). However, higher dimensionality results in lower computation efficiency.