C57BL/6J mice (9–57 weeks old, male and female; Charles River Laboratories Japan, Inc., Yokohama, Japan) were used. All the experiments were approved by the Institutional Animal Care and Use Committee of The University of Tokyo (P19-031). Animal care and treatment were performed in accordance with the guidelines outlined in the Guide to Animal Use and Care of The University of Tokyo.

Two, three, or four mice were placed in a white arena (32 cm × 32 cm × 28 cm) and their behavior was recorded for approximately 5 min using a video camera (HDR-CX720V or HXR-NX80, Sony, Tokyo, Japan) set at a height of 110 cm. The recording conditions were as follows: frame rate, 60 Hz; resolution, 1,920 × 1,080 pixels. Mouse behavior under dark conditions were recorded with infrared light and darkroom safe light (the illuminance of room was 2–5 lx). The videos are summarized in Supplementary Table 1. All videos were recorded during daytime (8:00–20:00).

Mouse contours in each frame image were annotated using VGG Image Annotator (Dutta and Zisserman, 2019) (version 2.0.8). Representative examples are shown in Figure 2A. A total of 203 images in video #1 and 51 images in video #2 were manually annotated and used to train and validate the tentative contour detection model. The contours in videos #3 and #4 were predicted using the tentative detection model. Of these, 400 mis-predicted images were selected and corrected manually. These 203 + 51 + 400 images were used training the final detection model. In videos #5, #6, and #7, every 60th frame image was predicted using the tentative detection model. These predicted contours were manually corrected and used as the human annotation for the test dataset.

This study adopted the Mask R-CNN pre-trained with the Microsoft Common Object in Context (COCO) dataset (Lin et al., 2014; He et al., 2017; Waleed Abdulla, 2017) to identify the mouse contours. We fine-tuned Resnet101 stage 4 and the following layers with 203 images and validated it with 51 images for 350 epochs. The hyperparameters were set to default values (Waleed Abdulla, 2017). The model trained for 298 epochs was used as a tentative detection model. The obtained contours were expanded for five pixels and simplified by Ramer–Douglas–Peucker algorithm, where ε = 1.5. The ε value was decided as the number of simplified plots were similar to that of human plots.

Four-fold cross-validation was performed to examine the optimal epoch to train the Mask R-CNN (Supplementary Figures 1B, C). The hyperparameters were set to default values (Waleed Abdulla, 2017). We surveyed all validation loss values for 350 epochs and calculated their mean values (Supplementary Figure 1B). Because the lowest mean value was recorded at 341 epochs, we fine-tuned the Mask R-CNN pre-trained with the COCO dataset for 341 epochs with the full training dataset. The trained model is used as the final detection model.

Contour regions were acquired by Mask R-CNN and buffered under the following conditions: dilation (10 iterations, 3 × 3 kernel) and Gaussian blur (7 × 7 kernel). The frame images were cropped along each buffered contour. The buffered images were resized to one-fourth size for reducing computational time and converted to grayscale. Then, the distance and sum of intensities for every pair of pixels in the images were calculated for each contour region (Figure 3A), and two-dimensional histograms were created (Figure 3B). Absolute values of the difference between the pair’s histograms were calculated and averaged to compare the similarities between mouse pairs. The most similar pairs were assigned the same ID. When the number of mice was n, n contours were assigned IDs and the others were ignored.

The training and prediction of neural networks and other calculations were conducted on a desktop computer equipped with an Intel Core i9-9900KS CPU, 64 GB RAM, and NVIDIA GeForce RTX 2080 Ti. Image processing and training of the neural networks were conducted using the Python.

In this study, we recorded top-view videos of multiple mice in an open-field arena. The following steps were used to track multiple mice (Figure 1). First, we divided the videos into frame images and identified the mouse contours in each frame image (detection step). Next, we assigned the ID to each mouse by calculating contour similarities (tracking step). Finally, sporadic misses in mice and unnatural predictions were detected and semi-automatically corrected (correction step).

In the detection step, video files were divided into frames, and individual mouse contour regions in each frame were independently detected using a Mask R-CNN (He et al., 2017; Waleed Abdulla, 2017). Because Mask R-CNN requires a training dataset to identify mouse regions, we had to annotate the contour of each mouse for hundreds of images (Figure 2A). However, such annotation processes are generally labor intensive and time consuming.

In this study, we introduced a stepwise method to reduce the annotation labor (Figure 2B). First, because utilizing models pre-trained with large-scale datasets, known as transfer learning, can efficiently reduce the necessary number of annotations, this study adopted the Mask R-CNN model pre-trained with the Microsoft COCO dataset (Lin et al., 2014) and fine-tuned it with the following simple dataset. The behavior of two C57BL/6 mice in an open-field arena was recorded using a video camera. The video files were divided into frames, and 203 and 51 frames were selected to train the Mask R-CNN and to validate its performance, respectively. During the training, loss values, which show the difference between Mask R-CNN predictions and human annotations, gradually decreased and reached a plateau at approximately 300 epochs (Supplementary Figure 1A). This trained model is hereafter referred to as the “tentative detection model.”

Next, using this tentative detection model, we predicted mouse contours in videos of three–four C57BL/6 mice. We found that some mice were missed and/or mis-plotted when one mouse contacted and/or occluded other mice (Figure 2C). We manually selected 400 mis-plotted frames and modified the contours. Finally, we trained the Mask R-CNN with the 203 + 51 + 400 dataset (details in section “Materials and methods” and Supplementary Figures 1B, C). The training loss values converged successfully (Figure 2D). This model was referred to as “final detection model.” The final detection model successfully predicted difficult images in which a mouse was missed by the tentative detection model (Figure 2C).

Next, we predicted the mouse contours in each video frame using the final detection model. Because the relationship of detected mice between frames remains unknown, assigning IDs to individual mice is necessary in every frame. First, the mice identified in the first frame were assigned unique IDs. We then calculated the similarities of the detected mice between frames using the color-correlogram-based method, as adopted in idTraker (Pérez-Escudero et al., 2014) (also see section “Materials and methods”), and assigned correct IDs to contours in each frame. Briefly, we calculated the distance and sum of intensities for every pair of pixels in the images of each counter region (Figure 3A) and created two-dimensional histograms (Figure 3B). Then, to compare the similarities between mouse pairs, absolute differential values between pair histograms were calculated and averaged. The values for all pairs were sorted in ascending order, and the corresponding IDs were assigned in order (Figure 3B).

We created videos that displayed the geometric centers of contours and assigned IDs for each mouse, and checked their predictive performance. There were two types of problematic predictions: (i) sporadic misses (Supplementary Video 1; mouse ID #3 at 0:00:02) and (ii) irreversible ID switches (Supplementary Video 2; mouse ID #0 and #1 at 0:00:00). Sporadic misses were defined as cases in which the final detection model underestimated the number of mice per frame. Irreversible ID switches were defined as cases where the assigned IDs were accidentally but persistently interchanged between individual mice. The proportion of sporadic misses in all frames tended to increase according to the number of mice in the videos, and an irreversible ID switch was found in one video (Table 1).

To address these problems, semi-automated processing was applied to predictions. First, sporadic misses were automatically fulfilled with previously predicted IDs (Supplementary Figure 2A, Video 3). Next, to identify the ID switches, the distances between the geometric centers of all pairs of continuous frames were calculated (Supplementary Figure 2B). When the distances between the coordinates of mouse ID i at frame N and N + 1 were not the shortest among those of all ID pairs, we set our method to offer “tracking warning.” Consequently, five tracking warnings were identified in the predicted videos (Table 1). One of the tracking warnings successfully identified irreversible ID switching, whereas the other three warnings were irrelevant to irreversible ID switching. Here, we manually exchanged IDs when switching occurred (Supplementary Video 4). These semi-automated correction steps were applied to the following predictions.

A test dataset was created to evaluate the performance of the proposed method. The contours of the videos of two, three, and four C57BL/6 mice were acquired using the tentative detection model. These contours were checked and corrected by humans every 60th frame and used as manually annotated contours.

Next, each video was predicted using the established method. In the correction step, 0.03–0.82% of the frames per video were compensated as sporadic misses. Additionally, three tracking warnings were proposed, although irreversible ID switches were not observed (Table 2). After the corrections, we evaluated the predictive performance by calculating the distances between the geometric centers of the predicted and manually annotated contours every 60th frame. All values were less than 1.5 cm, and 99.8% values were less than 0.5 cm (Figure 4A). As the body size of the recorded mice was approximately 8 cm (Supplementary Figure 3), these results suggest that our method precisely predicts the coordinates of geometric centers.

In addition, we examined whether our system could be applied to videos recorded under dark conditions that were not used for the training dataset. Videos of two, three, and four C57BL/6 mice in an open field arena were recorded with infrared light and predicted using the established method. In the correcting step, 0.79–4.37% of the frames per video were compensated as sporadic misses. and 42 tracking warnings were proposed, which identified irreversible ID switches twice (Table 2). These results were comparable to those of the videos of two and three mice recorded under light conditions. However, the predictive performance for the video of the four mice under dark conditions was slightly inferior to that under light conditions. Adding a training dataset of images recorded under dark conditions can improve performance. Representative performances under light and dark conditions after the corrections are shown in Supplementary Videos 5, 6.

Finally, we showed the typical methods to evaluate activities and social interactions using the predicted coordinates of the video recorded under light conditions. Our method enabled us to calculate the cumulative travel distances of the individual mice (Figure 4B). This might reflect the characters in social groups such as “quiet” and “restless.” In addition, sociality can be evaluated by calculating the distance between individuals and the duration during which the mice are located close together (Figures 4C, D). These analyses can reveal whether a mouse accompanies other mice. In conclusion, our established method can help us analyze the behavior of social groups.