Event-based Optical Flow on Neuromorphic Processor: ANN vs. SNN Comparison based on Activation Sparsification (2024)

2.1 ANN vs. SNN Comparison

Some key characteristics of ANN vs. SNN comparison works are shown in Table 1. From left to right, the items are:whether multiple forward-passes were required for an SNN prediction;how the SNN(s) was trained;whether the energy cost of memory access (reading weights, reading and writing neuron states) was considered;what techniques were adopted to optimize ANN inference;what processor was adopted for neural network inference;the maximum average number of spikes received per synapse for the SNN(s) to be more energy efficient than its ANN counterpart(s).The meaning of other items in the table are introduced as follows:“\faCheck&\faTimes” means the studied SNNs have different results;“-” means the item was not clearly stated;“\faCheck” or “\faTimes” following the maximum spikes data indicates whether the studied SNNs are more energy-efficient;“ANN-SNN Conversion” means the SNN(s) was converted from trained ANN(s);“Acti. Spars.” indicates that the activation sparsity of ANNs was exploited to reduce energy cost. When an activation is zero, all computations and memory accesses needed for the synaptic operation can be saved. The “*” on our work means that we not only exploit but also increase the ac./sp. sparsity.

It was shown in Dampfhoffer etal. (2022) that memory access is much more expensive than computation (>95% of total energy cost), thus using AC instead of MAC has a small impact on the overall energy cost.About the processors in Lemaire etal. (2022), 4 accelerators for sequential and paralleled processing are synthesized on FPGA. However, the SNN accelerator does not support parallel processing for convolution layers while the ANN accelerator does, harming the fairness of comparison.The highlight of Deng etal. (2020) is that it is the only work that studied tasks of event cameras. It was found that SNNs are less accurate and require more computation when the inputs are images and SNNs require multiple forward-passes. On the contrary, when the inputs are from event cameras and SNNs do a single forward-pass, SNNs are better in both accuracy and computation cost. The shortcoming is that only the number of operations was shown.Similarly, our work studied an event-based vision task where an SNN achieves better efficiency but, compared with Deng etal. (2020), we show more comprehensive experimental results based on accelerating the inference of both ANN and SNN.

2.2 Activation Sparsification

Exploiting and increasing ANN activation sparsity is mainly motivated by improving the network inference speed on CPUs.Authors of Yang etal. (2021) proposed to rank the channels of a feature map based on their activation magnitudes. Only channels with top rankings will participate in the computations of the following layers.Similarly, for the outputs of multi-layer perceptrons, all other than the largest $k$ (hyperparameter) entries are set to zero in Li etal. (2023).It was found that inducing activation sparsity improved training robustness to label noise and image noise, and led to better prediction confidence.Instead of sorting activations by magnitudes, the authors of Cao etal. (2019)proposed to skip the computations that are predicted to produce zero activations by a quantized version of the same network layer.Note that the works mentioned above Yang etal. (2021); Cao etal. (2019); Li etal. (2023) induced computational overhead to exploit activation sparsity. To avoid the overhead, our solution directly ignores activations smaller than the trainable activation thresholds that stay constant during inference.

As for increasing ANN activation sparsity,the training scheme proposed by Grimaldi etal. (2023) produces a set of individual pixels that are forced to have zero activations on all of their channels, leading to columns of the activation matrix skipped when using the im2col-based general matrix multiply (GEMM) technique and thus accelerates computation.A $L$1 loss that regularizes the activation map was adopted to punish big activation values and thus produce more zero and near-zero activations Georgiadis (2019).After fine-tuning with this regularizer, activation density was decreased to $\sim$50% of the original model and the accuracy slightly increased. Similarly, authors of Runwal etal. (2023) used the hyperbolic tangent (TanH) function with a scaling parameter to sparsify the activation maps of rectified linear unit (ReLU) and a differentiable approximation of the $L$0 norm for other activation functions.In Kurtz etal. (2020), Hoyer regularization Hoyer (2004) was applied to activations of a pre-trained network during fine-tuning.Besides, the forced activation threshold ReLU activation function (FATReLU) (Eq. (1)) was proposed to replace ReLU.Notably, FATReLU is not differentiable. Therefore, Kurtz etal. (2020) performed a manual binary search of desired FATReLU thresholds based on whether the network accuracy can be recovered by retraining after applying the thresholds.Significant GPU hours and human labor are required by the manual search, especially for deep networks with many layers.To avoid such big workloads, we propose an approach (Subsection 4.1) that makes the FATReLU thresholds trainable, requiring only a single training attempt.

3.1 FireNet Architecture

The network architecture we study is shown in Fig. 1. It is based on the FireNet in Hagenaars etal. (2021). The ANN version and the SNN version have the same architecture of six convolution (conv) layers and two recurrent convolution (rnn-conv) blocks.All conv layers are single-strided and have 3-by-3 kernels except the last layer for flow prediction, which has 1-by-1 kernels. All intermediate tensors have 32 channels.

For ANN, a conv layer has non-zero trainable biases.To have a fairer comparison with the SNN, we made two adaptations from the RNN-FireNet (with vanilla ConvRNN) in Hagenaars etal. (2021). Firstly, we use FATReLU Kurtz etal. (2020) activation function (to be introduced in Subsection 4.1) instead of ReLU to pursue higher activation sparsity. Secondly, we modified the vanilla ConvRNN block used by Hagenaars etal. (2021), shown in Appendix B. As shown in Fig. 1, our rnn-conv block (deployed in SENECA Core 2 and 5) has two conv layers instead of three, and the TanH activation function is replaced by FATReLU. This modification reduces the model size and results in sparse information instead of dense one remembered within the rnn-conv block. The computational demands for the recurrent conv layer (shown below the Tensor Sum icon) and the following layer are greatly reduced thanks to the sparse hidden state. Moreover, as to be shown in Subsection 5.2, our smaller-sized and sparser FireNet models have higher accuracy than the original versions reported in Hagenaars etal. (2021).The SNN FireNet in this work is the same as the LIF-FireNet in Hagenaars etal. (2021), all its conv layers have zero biases.

The presentation of the event camera measurements consists of per-pixel and per-polarity event counts, same as Hagenaars etal. (2021). It gets populated with consecutive, non-overlapping partitions of the event stream. For training, each input partition (referred to as an event frame) contains a fixed number of events (1,000). For testing, events within a certain time duration are accumulated in an event frame.

3.2 Processing Mechanism of SENECA

SENECA Tang etal. (2023b); Yousefzadeh etal. (2022) is a programmable digital neuromorphic processor designed with a scalable number of cores.Each SENECA core consists of a high-bandwidth SRAM data memory. In this work, we use two sizes for data memory, 256 Kilobytes and 2 Megabytes, for different image resolutions, 56$\times$56 pixels and 120$\times$120 pixels, respectively. The resolution is constrained by the size of data memory due to the need to store the neuron states.A SENECA core has 8 neuron processing elements (NPEs) that operate in a vector-like fashion. The NPEs are hardware functional units that are time-multiplexed to perform neuron activity computations, e.g. addition, multiplication, comparison of numerical values, read from memory, write to memory, etc. A SENECA core has a RISC-V controller to conduct other operations, e.g. decoding received event¹¹1In the context of communication between SENECA cores, an event refers to the inter-core message. An event has 32 bits, In this work, the directions of event flows are shown in Fig. 1. To distinguish, we refer to the measurements of event cameras as camera spikes., calculating the addresses of the neuron states and weights, encoding ac./sp. into events, etc. More details can be found in Appendix C, D and Tang etal. (2023b); Xu etal. (2024).

We map a conv layer or a rnn-conv block to a SENECA core, as shown in Fig. 1. Eight SENECA cores are cascadedly connected.In the processing of the first conv layer, the input camera spikes are pre-sorted in spatial order. Camera spikes that are closer to the upper edge of the frame are received earlier. For camera spikes on the same row, the ones closer to the left are received earlier.Once camera spike(s) at a pixel is received, the neuron states within the patch of pixels whose receptive fields cover the camera spike’s pixel location are located and updated accordingly. Conv layers of FireNet have 3-by-3 kernels, so a spike updates $3\times 3\times 32$ neurons in a 3-by-3 pixel patch. When the newly coming spike is on a different pixel, the program switches to the new set of neuron states according to the new pixel location.Because spikes come in spatial order, it is deterministic whether there will be possible future spikes falling in the receptive field of neurons at a pixel location.If not, it means that a pixel location is fully updated and thus the neuron states at this location can be input to FATReLU (for ANN) or compared with the firing threshold (for SNN).SNN spikes and ANN activations bigger than FATReLU thresholds will then be sent to the following layer in another SENECA core by SENECA events. Smaller ANN activations are ignored. Thus SENECA exploits ANN’s activation sparsity.Since the neurons are fired in spatial order, the ac./sp. sent to the following layer are in the same spatial order. The following layer in another core starts processing as soon as receiving the first ac./sp.. At the same time, the processing of the first layer is still ongoing, leading to parallelization of the layers/cores that requires less network inference time cost than single-thread processing.This processing mechanism is called event-driven depth-first convolution. It is illustrated in detail in Appendix D.

The rnn-conv block’s two conv layers are processed one after another. The processing of the recurrent layer (shown below the Tensor Sum icon in Fig. 1) starts after finishing the forward layer on the primary data pass (shown on the left the Tensor Sum icon in Fig. 1). It is possible to map the recurrent layer in another core to parallel the two conv layers of the rnn-conv block. The latency (required time to get all optical flow predictions) remains the same but it could reduce the time cost of the rnn-conv block to finish all processing. We leave it to future works.To reduce the number of memory-accessing operations, it is natural to perform data reuse. SENECA reuses neuron states by a mechanism called ac./sp. grouping.Ac./sp. at the same pixel location can be processed together because, in convolutional operation, they update the same set of neurons. We set the group size to four. Firstly, the neuron states are loaded from the memory. Then, four ac./sp. are integrated into the neuron states. Lastly, the neuron states are stored back in memory. In this way, the number of memory access operations required to process four events is reduced from four to one.If there are fewer than four ac./sp. at a pixel location, dummy zero ac./sp. fill(s) the group.In addition to neuron states, an ac./sp. is used by all 8 NPEs to multiply with 8 weight parameters and then update 8 neurons in 8 channels of a pixel location. Weights are not reused by our processing mechanism.

4.1 Trainable Thresholds for Activations (ANN)

4.2 Sparsification Regularizers (ANN and SNN)

4.3 Threshold Initialization based on Prior Activations (ANN)

Using the surrogate gradient for FATReLU thresholds and loss function shown in Eq. (2), an activation-sparse ANN can be trained from scratch by a single training attempt. In practice, we find that the initialization of FATReLU thresholds affects the network performance. Thus, we propose initializing the FATReLU thresholds based on the activation statistic of a trained network. Specifically, all thresholds have fixed zero values in the first training attempt, and the network is trained from scratch by $L_{flow}+\sum_{i}\lambda_{i}\cdot(\sum\mathrm{ReLU}(\mathbf{x}_{i}))$. Then, we run the trained network (inference only) on a subset of the training set to log the activation tensors. Based on the logged tensors, the initial threshold $T_{N,M,init.}$ for the $M$th channel of the $N$th layer is set as the median of all logged activations at the $M$th channel of the $N$th layer, i.e., the network is initialized to have 50% non-zero activations of the original trained network, statistically. The second training attempt fine-tunes the trained network with the initialized thresholds by Eq. (2).

5.1 Network Training

We train our optical flow FireNet based on the open-sourced code of Hagenaars etal. (2021). The regularizers for activation sparsification are added to the event deblurring loss for self-supervised learning of optical flow $L_{flow}$, as shown in Eq. (2). A reader who is interested in $L_{flow}$ and the training strategies can refer to Hagenaars etal. (2021).The network is trained for 100 epochs on the UZH-FPV dataset Delmerico etal. (2019). We use the AdamW optimizer with a weight decay of 0.01 and the OneCycleLR scheduler with a maximum learning rate of 2e-4.In our practice, networks are trained with different $\lambda_{s}$. It ranges from 1.0e-6 to 2.6e-6 for ANN and 1.6e-7 to 2.6e-7 for SNN. As for $\lambda_{i}$, the layer with the highest ac./sp. density has $\lambda_{i}$=2.0 while other layers have $\lambda_{i}$=1.0. The purpose is to reduce the difference in densities between layers to avoid one or several layer(s) becoming much denser than others and becoming bottleneck(s) of inference speed.

5.2 Network Accuracy and Activation Density

In this work, we focus on the lightweight FireNet architecture. But to show that the proposed activation sparsification approach can generalize, we also applied it to EV-FlowNet, using the code from Hagenaars etal. (2021). Note that, we do not intend to deploy EV-FlowNet on the neuromorphic processor due to its large size. So we do not modify the architecture of its recurrent block.In Table 2, we took two networks, EV-FlowNet (GRU) and FireNet (GRU), from Hagenaars etal. (2021) as the baselines of network prediction accuracy. The other networks in the table are trained by us. RNN-FireNet has 74,722 parameters. RNN-FireNet-S and RNN-FireNet-S-FT have 74,946 parameters. LIF-FireNet and LIF-FireNet-S has 74,818 parameters. RNN-EV-FlowNet has 23,531,752 parameters. RNN-EV-FlowNet-S has 23,535,240 parameters. LIF-EV-FlowNet and LIF-EV-FlowNet-S have 20,400,840 parameters.Surprisingly, our networks with vanilla RNN have better accuracy than the baselines with GRU. We attribute it to our training scheme with the AdamW optimizer and the OneCycleLR learning rate scheduler.

Ac./sp. density is calculated for the output ac./sp. tensors of the input layer and hidden layers. The ac./sp. density of each layer is logged for all testing samples. The average ac./sp. density of a network model is the average over all its layers and all testing samples. The output layer has a dense activation function, SoftSign for our adapted FireNet and Tanh for EV-FlowNet. The RNN blocks of EV-FlowNet have a dense Tanh activation function. We do not apply activation sparsification to such layers and do not take the 100% activation density into account when calculating the average activation density. The density of input camera spikes is not taken into account either, since it depends on the dataset and thus is not correlated to the network models.

In Table 2, we show two models of the sparsified ANN EV-FlowNet with vanilla RNN (RNN-EV-FlowNet-S) and the sparsified SNN LIF-EV-FlowNet (LIF-EV-FlowNet-S), respectively. “Smaller $\lambda_{s}$” and “bigger $\lambda_{s}$” mark two separate training attempts of the same network with the same ac./sp. sparsification approach but different weights for sparsification loss ($\lambda_{s}$).The accuracy and ac./sp. density of the two network models tell that ac./sp. sparsification with smaller $\lambda_{s}$ produces similar accuracy but higher ac./sp. sparsity than the network model without sparsification. Ac./sp. sparsification with bigger $\lambda_{s}$ can train network models with slightly lower accuracy but higher ac./sp. sparsity. Accuracy-sparsity trade-off is discussed in the next subsection.

As shown in the bottom half of Table 2, RNN-FireNet-S* has the highest accuracy. It is close to the accuracy of LIF-EV-FlowNet-S which has 20,400,840 parameters, $\sim$272.2 times of RNN-FireNet-S (74,946 parameters). The activation density of RNN-FireNet-S* is around half of RNN-FireNet. It indicates that lower density could exist together with higher accuracy. This phenomenon was also observed by works Georgiadis (2019); Kurtz etal. (2020); Yang etal. (2021); Hoefler etal. (2021); Runwal etal. (2023). We intuitively explain this phenomenon by that the sparsification regularizers can suppress the noise in activations. Only the activations carrying important information are kept. The “denoised” sparse activations can contribute to accuracy.RNN-FireNet-S-FT has only $\sim$13.3% activations of RNN-FireNet while maintaining a similar accuracy. For LIF-FireNet-S, we observe a small improvement in accuracy and 23.2% density of the LIF-FireNet trained without sparsification.In Section 6, RNN-FireNet-S-FT and LIF-FireNet-S are compared with hardware in the loop.

5.3 Ablation Study

To better understand the effects of the sparsification regularizers, we perform an ablation study.For simplicity, we use neuron density and pixel density to refer to neuron-wise ac./sp. density and pixel-wise ac./sp. density, respectively. Pixel density is the percentage of pixel locations that have at least one channel with ac./sp..For instance, an activation tensor of a conv layer has 5$\times$5 pixel locations and 8 channels on each pixel. If there are 16 non-zero elements in the activation tensor, then the neuron-wise activation density is 16/(5$\times$5$\times$8)=8%. If these 16 non-zero elements are located on 2 pixel locations (all 8 channels are non-zero for these 2 pixels), then the pixel-wise activation density is 2/(5$\times$5)=8%. If there are 16 pixels each of which has only one non-zero channel, then there are 16 activated pixels and the pixel density is 16/(5$\times$5)=64%. Therefore, pixel density reflects the spatial density of ac./sp. in the 2-dimensional domain of pixel locations, i.e. image plane.

The neuron density and average endpoint errors (AEEs) of network models trained with different settings of sparsification loss are shown in the left subplot of Fig. 2. For simplicity, ANN represents RNN-FireNet and LIF represents LIF-FireNet.To remove as much of the effect of randomness in network training as possible, we trained $\sim$10 models for each loss setting with different weights on the sparsification loss ($\lambda_{s}$ in Eq. (2)). In the left subplot of Fig. 2, if the point of Model A is on the lower left side of the point of Model B, it means Model A is better in both accuracy and neuron sparsity. In this case, if the two models are trained with the same loss setting, Model B is omitted in both subplots.

First, we discuss the information in the left subplot. For ANN, when applying the $L$1 regularizer on neuron membrane voltage alone (ANN-S(vol.,ReLU), green inverted triangle), the accuracy significantly increases and the neuron density decreases to around 50% of the networks trained without sparsification loss (ANN (ReLU), orange circle).Comparing ANN-S(vol.,FATReLU) marked by the red crosses and ANN-S(vol.,ReLU), we can notice that having FATReLU with trainable thresholds brings small improvement to accuracy and the density is little affected.After applying the $L$2 regularizer that encourages bigger thresholds, density dramatically drops to $\sim$5%, as shown by ANN-S(vol.&thre.,FATReLU) marked by the green diamonds. Comparing the green diamonds with the red squares (ANN-S-FT), we can see that the latter performs better in both metrics, to a small extent.Similarly, for SNNs with LIF neurons, applying sparsification loss noticeably benefits both density and accuracy. A blue pentagon (LIF-S(vol.)) and an orange triangle (LIF-S(vol.&thre.)) are closely located, indicating the small effect from the $L$2 loss for the SNN firing thresholds.Then, we move to the right subplot of Fig. 2 to analyze the pixel density.As shown, SNNs have lower pixel densities ($\sim$39%) than ANNs. ANNs with high accuracy (AEE$<$4.0) have high pixel density ($>$68%).

6.1 Single Layer Comparisons in Controlled Conditions

Before testing the whole network on real data of the test set, we conduct a comparison to study how pixel density and number of ac./sp. on the same pixel affect the processing cost of a single conv layer. The results are shown in Fig. 3.In these experiments, the spatial resolution is 16$\times$16, 256 pixels in total. Each pixel has 32 channels, the same as FireNet. We consider the total processing cost of receiving SENECA events, updating neuron states, and comparing the neuron states with the thresholds. Capturing, encoding, and sending the ac./sp. are disabled because it is hard to control the number of output ac./sp..In the first set of experiments (top row of Fig. 3), we generate one ac./sp. for each pixel because most of the output tensors of FireNet layers have a single ac./sp. per pixel, as shown in Fig. 4. The numbers of pixels that have an ac./sp. are 0, 50, 102, 151, and 204.Shown in the top two subplots of Fig. 3, there is a clear linear correlation between the number of pixels having ac./sp. and the inference time. SNN is more time-consuming than ANN because the decoding schemes of SENECA events are different between SNN and ANN.

For ANN, we use a 32-bit event to encode an activation, 16 bits for the channel index (unsigned integer), and the rest 16 bits for the activation value (BFloat16). In contrast, for SNN, we use a bit coding scheme. The 32 bits of an event encode the 32 channels. If a channel has a spike, the corresponding bit is 1 otherwise 0. While for ANN, the number of events is the same as the number of activations. SNN requires fewer events and thus is more efficient in terms of inter-core communication.However, to decode the spikes from the 32-bit event, 32 iterations are required to check every bit. When there is one ac./sp. at a pixel, ANN takes 2.52$\mu s$ to decode. In contrast, SNN needs 5.14$\mu s$. When the number is 28, SNN takes less time (8.04 vs. 8.14). This phenomenon tells us that the software implementation onboard SENECA should take into account the spike distribution of the certain network to select the better event encoding scheme. In this work, all experiments of SNN are conducted with the bit coding scheme.To eliminate the effects of event encoding schemes, we subtract the time difference for the SNN assuming it uses the same encoding as ANN, shown by grey circles in the top row of Fig. 3.After the subtraction, the SNN is more efficient. We attribute it to the fact that a binary spike does not require the memory access and multiplication required by an activation.Note that we use the power measurements of the SNN with the bit encoding scheme when calculating the energy cost. Using ANN’s encoding scheme should lead to lower power cost and thus less energy than what the top-right subplot of Fig. 3 shows.

As shown in the two bottom subplots of Fig. 3, increasing the number of ac./sp. from 1 to 4 leads to less growth in time and energy cost than increasing it from 4 to 6. It is because of the ac./sp. grouping strategy that processes four spikes together, introduced in Subsection 3.2. When the number is below 5, it requires only one neuron state updating operation. When the number increases to 6, it requires two operations thus the cost increases significantly. When the number is 10, three operations are required.Comparing the data points in the circles in the two subplots on the left, the points in the green circle have 204 ac./sp.. The points in the yellow circle have 604. Although having more ac./sp., the time costs of the points in the yellow circle are less, due to the ac./sp. grouping mechanism that reduces the required numbers of memory access operations of neuron states. Therefore, higher neuron density does not necessarily mean more time cost, the spatial distribution of ac./sp. is an important factor. SENECA processing can be more efficient when the ac./sp. are concentrated at fewer pixels.

6.2 Spatial Distributions of Ac./Sp.

After the experiments of ac./sp. distribution of a single layer in controlled conditions, we compare and analyze the ac./sp. distributions of the ANN and the SNN on the real test set, and then deduce which network type can be more efficient.The boxplots reflecting the distribution of pixel density are shown in the left subplot of Fig. 4.ANN has a higher pixel density than SNN in 6 out of 7 layers. For the last 3 layers, almost all inferences have almost 100% pixel density.In the right subplot of Fig. 4, we can see that the SNN statistically has more spikes per pixel than the ANN. It is reasonable because the neuron densities of the ANN and SNN are similar and the SNN has lower pixel density. Based on Fig. 4, we claim that the SNN is more likely to have higher efficiency because its spikes are concentrated at fewer pixels. As discussed in Subsection 6.1, this feature benefits processing efficiency.

Fig. 5 visualizes the accumulated camera spikes (i.e. event frame that is the network input), ac./sp. maps produced by each layer, and event-based optical flow predictions. The caption for Fig. 5 introduced each type of its subfigure in detail. From Fig. 5, we notice that ANN has higher pixel-wise activation density, especially for Layers 5, 6, and 7. Almost every pixel has at least one activation while the number of activation(s) at one pixel is small, as indicated by the dark grey color in the first and third rows of Fig. 5 that correspond to the ANN’s activation maps. In contrast, Layers 5, 6, and 7 of the SNN produced many pixels without any spike. Most of such dark pixels are in the image regions where there is no input camera spikes. For image regions with many camera spikes, all layers of the SNN produce dense spikes.It can be summarized that higher contrast in an ac./sp. density map means a higher degree of spatial concentration of ac./sp. and higher pixel sparsity. The spike density maps of the SNN have higher contrast than the activation maps of the ANN. Thus SNN has higher pixel sparsity. This observation is consistent with Fig. 4.

Fig. 4 and Fig. 5 show the pixel-wise ac./sp. density of the FireNet architecture.One feature of FireNet is that the output tensors of all layers have the same spatial size, i.e., there is no downsampling or upsampling. For EV-FlowNet, a more complicated UNet-like architecture with downsampling encoding and upsampling decoding, we got similar results.The RNN-EV-FlowNet-S (4th row in Table 2) has average (over the testing dataset) pixel density (%) at each layer: [100, 85.39, 100, 100, 99.99, 69.1, 71.65, 65.78, 99.99, 100, 99.98, 100].In contrast, the layer-wise pixel density (%) of the LIF-EV-FlowNet-S (7th row in Table 2) is: [57.97, 63.69, 79.77, 90.49, 88.09, 70.05, 89.26, 85.07, 89.23, 85.25, 79.54, 67.89]. For 9 out of 12 layers, the ANN (RNN-EV-FlowNet-S) has higher pixel density than its SNN counterpart (LIF-EV-FlowNet-S).

6.3 Network Experiments

Recalling Subsection 3.2, we test two resolutions on two sizes of on-chip memory of SENECA.We first show and discuss the time and energy costs of the event frames with 56$\times$56 pixels and then the results of 120$\times$120 pixels. The network prediction accuracy of 120$\times$120 pixels onboard SENECA follows.For each type of network, we select three event frames from the test set that lead to different neuron densities. For each layer of the SNN, the frames lead to respectively $\sim$20%, $\sim$100%, and $\sim$200% of the average neuron density of the layer over the test set. For the ANN whose neuron density is narrower distributed, the three input frames respectively lead all layers except the 6th layer to have $\sim$50%, $\sim$100%, and $\sim$150% of the average layer-wise neuron density.Although the 6th layer does not coincide with expectation, the two selected frames for $\sim$50% and $\sim$150% density have the least differences from the expected densities among all frames in the test set.

Fig. 7 shows the time cost and its correlations to neuron density and pixel density. Same as the results shown in Subsection 6.1, we see a positive correlation between pixel density and time cost. The three frames of the ANN have similar time costs because ANN’s last three layers have almost 100% pixel density on almost all testing frames (shown in Fig. 4), including the selected three frames. These layers took longer time than other layers due to the high pixel density. Because of the parallelization of the layers/cores introduced in Subsection 3.2, those layers became the bottlenecks of the whole network inference. Given the similar pixel densities ($\sim$100%) of the bottleneck layers of the testing frames, it is reasonable to get similar time costs.As shown in the right subplot of Fig. 7, when the pixel density is $\sim$70%, the SNN has a larger latency than the testing frames of the ANN. The reason is that the SNN testing frame has noticeably higher neuron density, as shown in the left subplot.Recalling the two bottom subplots of Fig. 3, more ac./sp. on the same pixel leads to longer time.

Comparing the data points corresponding to the average neuron densities (the one in the middle for each polyline), it is noticeable that the SNN has higher time efficiency. The SNN costs 44.9ms in processing the selected testing frame, which is 62.5% of 71.8ms, the time cost of the ANN.The SNN testing frame for average neuron density leads to pixel density (%) [9.3, 25.2, 45.3, 30.8, 31.5, 55.4, 56.9, 59.2] for each layer, 9.3 is the density of the input camera spikes and the average density of the 7 layers is 43.5%. It is higher than the average pixel density (%) over the whole test set, which is [8.6, 22.2, 40.0, 28.9, 30.7, 51.6, 50.0, 51.3], 8.6 is the density of the input camera spikes and the average density of the 7 layers is 39.2%. Therefore, the average time cost of the SNN over the test set could be lower than the selected testing frame.As for the ANN, the testing frame for average neuron density results in pixel density (%) [8.8, 28.6, 82.5, 37.4, 22.8, 96.2, 98.3, 99.6] for each layer, 8.8 is the density of the input camera spikes and the average density of the 7 layers is 66.5%. It is slightly lower than the average over the whole test set, [8.6, 30.9, 82.5, 38.5, 25.1, 94.9, 98.8, 99.9], 8.6 is the density of the input camera spikes and the average density of the 7 layers is 67.2%.Therefore, the ANN could cost more time than the selected frame, on average over the test set.Based on the fact that the SNN’s time cost is 62.5% of the ANN’s when processing the selected testing frames for average neuron density, we claim that the SNN is averagely more time-efficient than the ANN on the test set and the average time cost of SNN is less than 62.5% of the ANN’s.In addition, SNNs can be more time efficient if using the same SENECA event encoding scheme as ANN, as discussed in Subsection 6.1.The energy costs are shown in Table 3. The SNN’s energy cost is 75.2% of the ANN’s.

The event frames with 56$\times$56 pixels are downsampled from the central patch of 112$\times$112 pixels. It only covers 19.14% area of the original event frame with 256$\times$256 pixels. Besides, we find that, for many frames, there is no available ground truth optical flow for any pixel. Therefore, we only evaluate the network accuracy on event frames of 120$\times$120 pixels on SENECA. For each of the ANN and the SNN, we select a frame producing $\sim$100% of the average neuron density over the whole test set on all layers.The statistical accuracy and density of the whole test set shown in Table 4 are obtained by the Python-based SENECA simulator running on a CPU server. This simulator produces exactly the same computation results as SENECA without any precision difference.SENECA uses 16-bit BFloat16 as the data type, but the network parameters trained on GPU are 32-bit single-precision floating-point numbers. We do post-training quantization that rounds the trained network parameters to BFloat16 values to deploy the networks on SENECA.We also run the same tests on GPU to compare with the results from SENECA.In general, SENECA produces slightly better accuracy and sparsity than GPU, which means that the post-training quantization to BFloat16 does not deteriorate accuracy. Intuitively, we propose the hypothesis that BFloat16 has enough precision in the context of optical flow prediction using FireNet.

The time and energy costs of processing 120$\times$120 pixels are listed as follows. The latency and total time of the ANN are 182.69ms and 326.39ms, respectively. For SNN, the measured durations are 148.91ms and 225.77ms, respectively. The SNN is $\sim$18% and $\sim$30% more efficient in latency and total time cost, respectively.The energy cost of the ANN is 5753.4 $\mu$J and the SNN consumes 4174.4 $\mu$J. The SNN has a $\sim$27% advantage.In general, the advantage of the SNN over the ANN observed in the test on 120$\times$120 pixels is similar to the observation in the test on 56$\times$56 pixels.