Due to light scattering from different focal planes, the high-frequency structure of the spatial information is significantly suppressed. Therefore, we used a high-pass spatial filter (Pnevmatikakis and Giovannucci, 2017; Zhou et al., 2018) to enhance the spatial features and contrast. The kernel was generated from a Gaussian filter,

where h(x, y) is a Gaussian convolution kernel used for background removal and (x₀, y₀) is the center of the kernel. The standard deviation of the Gaussian kernel is the width of neuron. For the data captured by the UCLA miniscope, we observed that the diameter of a neuron in the FOV is usually around 10 pixels. Based on empirical evidence, the size of the kernel should be three times the size of the neuron. Then, the kernel subtracts its spatial mean value h̄(x,y) to generate a new kernel h~(x,y) for filtering. In addition to removing the blurred background and increasing the image contrast, the component of the filter kernel with a positive value approximates the size of the cells, thus retaining the neuron shape and calcium fluorescence during the subsequent signal extraction.

To efficiently calculate the similarity between each frame and the template during frame alignment, we adopted the method proposed by Padfield (2010), which reduces the computational complexity by calculating the normalized cross-correlation (NCC) in the Fourier domain. Assume a reference image f₁(x, y) and a moving image f₂(x−u, y−v), with f₂ shifted by (u, v). Then, the spatial form of the NCC can be defined as

where D(u, v) is the overlap region of the two images and f1,u,v¯ and f2,u,v¯ represent the mean intensity of f₁ and f₂ in D(u, v). Since the cross-correlation of two images can be calculated in the Fourier domain by CC(f1,f2)=ℱ−1(F1·F2*), we can also infer the components in Equation (3) in the Fourier domain. The two images can be represented as F₁ = F(f₁) and F2*=ℱ(f2′), where F(·) represents the FFT operation, F2* is the complex conjugate of the Fourier transform and f2′ is f₂ is rotated by 180°. In addition, assuming that i₁ and i₂ are images of ones with the same size as f₁ and f₂, respectively, we can define I₁ = F(i₁) and I2*=ℱ(i2′). Based on the results of Padfield (2011), the Fourier version of the NCC can be represented as

The motivation for using this approach is that computations in the Fourier domain have lower order complexities (O(n·log(n))) than those in the spatial domain (O(n²)), where n indicates the number of pixels in the computation. In addition, because of the normalization, NCC can be invariant to the linear gray value changes (Emmenlauer et al., 2009; Yu and Peng, 2011), which can reduce the influence of the illumination changes caused by the mice movements, out of focus brightness as well as wires poor contact at different time steps in the real-time experiments.

The displacement estimated by the NCC only has pixel-level accuracy. To further improve the accuracy, subpixel registration is necessary. The most commonly used subpixel registration methods include upsampling with different interpolation methods and curve fitting (Shimizu and Okutomi, 2002; Debella-Gilo and Kääb, 2011). Although some upsampling methods may be more accurate, the number of pixels used in the computation will be greatly increased, whereas the parabolic curve fitting method can be computed more efficiently. In parabola fitting, the matching integer pixel point (x₀, y₀) can be obtained through the NCC map. The subpixel location can be calculated by independently fitting a one-dimensional quadratic function based on the position with the highest correlation and its adjacent location in the NCC map: (x₀ − 1, y₀) and (x₀ + 1, y₀) in the X-direction and (x₀, y₀ − 1) and (x₀, y₀ + 1) in the Y-direction. The subpixel position can be estimated by Equations (5) and (6):

where NCC(x₀, y₀), NCC(x₀ − 1, y₀), NCC(x₀ + 1, y₀), NCC(x₀, y₀ − 1) and NCC(x₀, y₀ + 1) indicate the integer matching locations and their adjacent points in the NCC map. Finally, the offsets can be estimated by adding the detected subpixel displacements to the integer offsets.

To quantitatively analyze the accuracy, we generated a simulated video by shifting a single frame 5,000 times. The artificial offset at each time step was random and within 10 pixels. Then, we investigated the impact of applying a high-pass filter on the registration precision. Next, we used OpenCV template matching, rigid NoRMCorre and RTMC to detect offsets in the simulated video and calculated the deviations for statistical analysis. In this step, we focused on evaluating how each algorithm detected random rigid translations.

Although simulated data can provide the ground truth for quantitative analyses of registration performance, this data does not reflect all the characteristics of the calcium imaging data. Therefore, we also collected single-photon calcium data from the mouse motor cortex in vivo. We used the mean (CM) metric (Pnevmatikakis and Giovannucci, 2017; Tegtmeier et al., 2018) to quantify the motion correction performance of the different algorithms on real data. The CM metric is based on the correlation coefficients (CC) between the mean image of a video and each individual frame after motion correction. A higher CC score indicates a better match between the current frame and the template; thus, this metric can be used to compare the performance of different algorithms at the single-frame level. In addition, to eliminate the influence of different interpolation methods on the CC score when a frame is translated, all the interpolation methods used in the testing approaches were bilinear interpolations. Prior to performing the analysis, we removed the black border generated by shifting the frames during registration.

All the equipment and surgical procedures used for data acquisition were described in our previous study (Wang et al., 2019). In brief, the UCLA Miniscope was used to capture the data, and a 1.5 mm-diameter glass coverslip was pressed against the brain tissue through a skull hole. The imaging data were acquired at a rate of 20 frames per second with custom DAQ software.

First, to illustrate the importance of motion correction, we manually labeled some neurons. The spatial calcium fluorescence intensity of each observed neuron was modeled as the neural spontaneous fluorescence plus the background fluorescence fluctuation (Pnevmatikakis et al., 2016). In single-photon calcium imaging data, the background can be decomposed into the global background and the local background, with the local background fluctuation more noticeably affecting the observed neuronal fluorescence (Zhou et al., 2018). Therefore, the neural calcium signal was calculated by subtracting the mean value of the region covered by the neuron from the fluorescence value of the local background, where the local background was an annular region with a radius slightly larger than the neuron. Thus, small differences in the neurons before and after registration could be compared. Furthermore, in most cases, neuroscientists use deconvolution to extract the spike activities from the calcium fluorescence traces. In our experiment, we used OASIS (Friedrich et al., 2017) to deconvolve the traces to illustrate the influence of motion correction on spike extraction.

To further demonstrate the performance of RTMC in real-time experiments, we designed a real-time neuronal calcium event detection task. When the calcium signal exceeded a specific threshold, such as 3 standard deviations from the fluorescence intensity of the target neuron, and then returned to this threshold, a calcium event was recorded (Yaksi and Friedrich, 2006; Mukamel et al., 2009). For real-time experiments, we used a prerecorded video to label the target neurons and generate the thresholds. After collecting all the data for the real-time experiment, we processed the recorded videos with MIN1PIPE and determined the number of calcium events of the labeled neurons using the same method, then compared the results with those of the real-time experiment. MIN1PIPE is an offline processing pipeline that uses a neural enhancement function to remove the background of each frame and a deconvolution function to denoise the traces before and after motion correction. As a result, MIN1PIPE can protect the calcium traces from the effects of fluctuating background fluorescence and noise. The MIN1PIPE result can be regarded as the offline ground truth. In addition, during the experiment, we counted the number of motion-corrected frames output from the GPU and computed the equivalent output frame rate. In theory, if the system has no delay and motion correction is used, the equivalent output frame rate should be equal to the acquisition frame rate of the scope.

We first investigated the impact of high-pass filtering on motion correction. Light scattering from outside the focal plane blurs the whole background (Figure 2A), while high-pass filtering can produce visible textures such as mark points (Figure 2B). When calculating the similarity between each frame and the template during motion correction, the fuzzy background causes relatively high correlations in the region around the matching point in the NCC map (Figure 2C). In contrast, the NCC map has a more concentrated peak when using filtered frames (Figure 2D). After verification with the simulated data, the deviations in the frame offsets calculated after high-pass filtering can be as low as 0.1 pixels, while those calculated using raw frames for registration can reach 1.2 pixels (Figures 2E,F, p < 0.001, Wilcoxon test, n = 500 frames).

Next, we used the OpenCV template matching, rigid NoRMCorre and RTMC algorithms to register the simulated data to compare the precision of the three methods on single photon data. Figure 3A shows that RTMC has the smallest error and performs significantly better than the other two methods (RTMC vs. OpenCV, p < 0.001; RTMC vs. NoRMCorre, p < 0.001, Wilcoxon test, n = 5,000 frames). In terms of detecting offsets, RTMC produces a more accurate match between the frames and the template. The same result can be observed from the histogram of the deviations (Figure 3B). The deviations of RTMC and NoRMCorre were less than 0.2 pixels, while the error distribution of the OpenCV-based approach ranged from 0 to 1.2 pixels.

Since the simulated data only reconstruct the offsets in the FOV, we also needed to evaluate the accuracy of our method with real data. Figure 4 demonstrates the CM metric of the OpenCV template matching, rigid NoRMCorre and RTMC algorithms on three 5,000-frame mouse motor cortex videos from layer 2/3 and layer 5 of different animals. RTMC performs consistently well on datasets with different spatial characteristics such as blood vessels, illumination distribution, exposure intensity, etc. (Figure 4A). Figure 4B shows the CM metric for a subset of frames in each dataset; the OpenCV-based method had the lowest correlations for almost all the frames, while the other two methods had very close correlations. The scatter diagrams (Figures 4C,D) show the same trend: the correlation coefficients of RTMC were not significantly different from those of NoRMCorre (p > 0.05, Wilcoxon test, n = 5,000 frames), while both significantly exceeded those of OpenCV template matching (p < 0.01, Wilcoxon test, n = 5,000 frames) on nearly every frame. By observing the video (Supplementary Video S1) after motion correction, we found that there are still obvious motion residues in the FOV corrected by OpenCV template matching, whereas no visible motion artifacts in the videos corrected by the other two methods. In contrast to the results on the simulated data, there was no significant difference between the CM scores of RTMC and NoRMCorre. This is most likely because the pixel-shift difference between the two methods was approximately 0.2 pixels, which had little effect on the overall FOV. Moreover, fluctuations in the neuron population activity and local background fluorescence can lead to differences in the spatial features of the frames and the template. For the CM metric, the correlation between the frames and the template is more sensitive to changes in spatial features with hundreds of pixels than the difference between the 2 well-behaved methods. In addition, we also tested the effect of buffer size for updating template. We tested different buffer sizes ranging from 50 to 200 and found no difference in the accuracy of motion correction (Supplementary Figure S1). Therefore, we can set this parameter to 200, the same as the value used in CaImAn.

When applied to a freely-moving mouse, RTMC can improve the quality of the extracted traces (Figure 6). The calcium traces of the labeled neurons with motion correction (Figure 6A) have larger amplitudes and lower noise values than the traces extracted without motion correction (Figures 6B,D). These traces were calculated based on the mean intensities of the areas covered by the labeled ROIs, and these amplitudes are reduced when the actual neuron coverage area deviates from the labeled area. When the deviation between the two coverage regions changes frequently, the calculated fluorescence fluctuates frequently, resulting in noise. The movement-induced fluctuations in fluorescence also increase the energy of the traces, which is reflected in the power spectra and is consistent with the results of Victoria (Griffiths et al., 2020) (Figure 6C). Furthermore, since measurable calcium fluorescence is induced by neural action potentials (or spikes), the extracted traces influence the detection of action potentials (or spikes). Therefore, the transient response in the calcium traces can be evoked by the movement artifacts in the FOV, resulting in additional false positive spikes. After thresholding with 3 standard deviations of the spike activities, the inferred spikes in the motion-corrected traces are sparser and more concentrated (Figure 6E), which is consistent with the calcium dynamics model proposed in OASIS. The total number of spikes in the traces (mean±SD=118±44.6, n=15) was significantly less than that in the uncorrected traces (152±32.9, n=15), indicating that movement-induced false positive spikes were highly suppressed (Figure 6F). Therefore, the traces acquired by RTMC more accurately reflect the calcium fluorescence induced by neuronal spikes.

For real-time calcium event detection, RTMC also has high performance. Figure 7 shows the results of RTMC on a real-time calcium event detection task with a target neuron (Figure 7A). The real-time detection results were the same as those of offline detection extracted by MIN1PIPE (Figure 7B). Moreover, at a frame rate of 20 Hz with motion correction, the cumulative received frame number at each 5 s timestep was nearly equal to the theoretical number, despite the occasional frame drop (Figure 7C). After linear fitting, the growth rate of the received frame number was 20 frames per second, which is consistent with the acquisition frame rate, ensuring that RTMC can align frames during real-time data acquisition. Further analysis of these results revealed that all the traces in the 5 sessions of the 2 scenarios had similar shapes, including the peaks in each session (Figure 7D). Furthermore, the onset times of the calcium events extracted by RTMC matched the offline ground truth (Figure 7F). However, the directly extracted traces during real-time acquisition had more noise due to background fluctuations, resulting in a lower peak signal-to-noise ratio (PSNR) (Figure 7E), while MIN1PIPE dramatically improved the PSNR with its complex signal enhancement and deconvolution algorithms. In other words, although RTMC provides a real-time signal extraction approach for single-photon calcium data, motion correction is not the only factor that can influence the calcium signal quality. For example, background removal and deconvolution can improve the signal quality before and after signal extraction. To obtain higher PSNR signals, the above preprocessing and postprocessing are necessary, which is another challenge for real-time calcium signal extraction.