The data (Table 1) was split into train-validation-test sets according to a 70%-10%–20% ratio, and per video rather than on a frame-level. Since frames from the same video likely resemble each other, they should not be distributed across different subsets, as this could hinder the ability to determine whether the detector is overfitting. Additionally, we balanced day/night, bars/glass and single/multiple canines across the different subsets.

We compared a variety of object detectors for which we selected the best architecture and training parameters on the training and validation datasets: YOLOv2 (Redmon and Farhadi, 2017), YOLOv3 (Redmon and Farhadi, 2018) YOLOv4 (Bochkovskiy et al., 2020), YOLT (Van Etten, 2018), D-YOLO (Acatay et al., 2018) and ResNet-YOLO (Ophoff et al., 2022).

Our most performant detector YOLOv2 trained with the Complete Intersection-over-Union (CIoU) loss introduced in (Bochkovskiy et al., 2020) (Results 3.3.1), was fine-tuned from ImageNet pretraining and optimized with stochastic gradient descent. A cyclic learning rate was used ranging from 2 × 10⁻⁸ to 10^–3. The training was conducted with an effective batch size of 32 over a maximum of 30 epochs. We selected the best model on the validation set after training. The loss function employed the Complete Intersection-over-Union (CIoU). The input images were resized to 640 × 384 pixels. The input data was normalized using ImageNet statistics and augmentation included random horizontal flipping, color jitter and geometric jitter.

To optimally select the correct detections and assign the correct animal ID, we developed the following strategy. The detector renders several potential detections within a frame (or across multiple camera views) which are all processed by the harness classifier model. To select the optimal combination, we utilized the confidence scores from both the detector and the harness classifier. If a canine with the same ID was detected in the previous frame, we included the IoU score between the bounding box in the previous and current frame to further refine the selection of bounding boxes (Figure 2). Using this information, we calculated a score for each potential detection in the frame using the following formula: s c o r e = λ d C d + λ r C r + λ I o U I o U λ d + λ r + λ I o U C d and C r represent the confidence scores of the detector and Harness classifier model, respectively, and range from 0 to 1. I o U is the score between the current bounding box and the bounding box in the previous frame. When there is no previous detection, this score is set to 0. λ d , λ r and λ I o U are weighing factors for the corresponding scores. We achieved the best results with respective values of 1.0, 0.75, and 1.0.

In this way, each detection is given a score between 0 and 1 for every ID, with a higher score indicating greater certainty in our models' predictions. To filter out less certain detections and reduce the likelihood of false positives, only detection/ID combinations with a score >0.5 were retained. We subsequently used a modified Jonker-Volgenant algorithm (Jonker and Volgenant, 1987; SciPy, 2008; Crouse, 2016) to find the combination that yields the highest overall score. A significant advantage of this algorithm is that it prevents the same ID from being detected twice. In addition, because the algorithm runs on every frame, a misclassification on a certain frame does not influence the misclassification probability in subsequent frames.

As camera views of adjacent kennels are processed together, the same approach was applied in group-housed conditions when animals were able to move across different camera views (Figure 2). The Harness classifier model processes every detection across all different camera views, and the modified Jonker-Volgenant algorithm continues to select the best combination, thus preventing duplicate IDs across the various camera views. Another advantage of this simultaneous processing is that the algorithm can compare all detections and assign IDs based on its certainty. For example, if only two of three harnesses are clearly visible, the algorithm can correctly assign those two IDs and infer the ID of the third animal, even though the harness itself is not visible.

Finally, we used a Kalman filter (Kalman, 1960) to track the bounding boxes over time, smooth out abrupt changes and fill in missed detections.

We evaluated the performance of the tracking module on a collection of video snippets covering day and night with animals either mostly separate (Day/Night-S) or clustered (closely together with overlapping bounding boxes, Day/Night-C) (Table 1).

For the AI activity tracking, in postprocessing, the Euclidean distance (in pixel) was determined between the center of the current bounding box and the center of the bounding box on the previous frame/of the previous detection (Figure 4a). In this way, a measure of movement was obtained for every animal on every frame. These movements were then binned per time frame (generally per minute) to obtain data for visualization in “actigrams” that provide a detailed plot of activity levels over time per animal.

To get a more general overview of the animals’ activity, the movements described above were classified on a frame-level as ‘mobile’ or ‘immobile’ based on a cutoff (Figure 4a, and Methods 2.5.2). Mobile refers to animals moving from one point in space to another, while immobile refers to animals remaining in the same location.

We considered five poses: Lying, Sitting, Standing, Standing Up (on hind legs) and Standing Down (on front legs, e.g., when jumping off the bench). The initial dataset was split with similar ratios of the different poses in the subsets.

ResNet-18 (He et al., 2016) served as a backbone for the pose classifier, trained using the cross-entropy loss for up to 50 epochs using a batch size of 64. Again, the best model was selected based on its performance on the validation set. SGD and a cyclic learning rate (varying between 1 × 10⁻³ and 2 × 10⁻²) were used for optimization. The inputs were letterboxed (using gray to pad the borders) and resized to 256 × 256 pixels, normalized, augmented with random horizontal flipping, color jitter. We used class balancing during training, as well as detector-generated bounding boxes to make the model robust to imperfect bounding boxes.

The behavior classifier receives sequential cut-outs of a single canine over time from the tracker module. Unlike previous models, which primarily process spatial information, this model also incorporates temporal information by analyzing sequential frames. This additional temporal context is crucial, as behaviors are often not apparent from a single frame alone.

We considered three behaviors: eating, drinking and none (no behavior of interest) (Table 1). As animals can be provided with food from either a hopper or a bowl, video material from both conditions was subdivided into respective subclasses. This subdivision into more homogeneous classes increased accuracy. During postprocessing, both subclasses were recombined into a single eating category. Additionally, we included several challenging not drinking and not eating fragments in which animals were standing near or sniffing the drinking nipple or food hopper/bowl without actually drinking or eating. This was done to help the model learn to distinguish between similar appearances more accurately.

The data was split into “training-validation-test” according to a 60%-15%–25% ratio with balanced day/night and bars/glass. Since the behavior model operates on entire videos and individual videos cannot be divided across different data sets, these ratios were approximate at the frame level. To address the class imbalance in the dataset during training, we employed random weighted sampling.

We fine-tuned a small Vision Transformer (ViT-S/16) with joint space-time attention (Dosovitskiy et al., 2020; Arnab et al., 2021; Tong et al., 2022), initialized from a VideoMAE checkpoint, pretrained on Kinetics-400(Tong et al., 2022), as our behavior classifier. Inputs were 16-frame clips at 224 × 224, sampled with a temporal stride of 4, yielding a 64-frame window (∼2.5 s at 25fps). Training used a AdamW optimizer with a cosine decay learning rate schedule (base 1 × 10⁻³, minimum 1 × 10⁻⁶), weight decay of 0.05 and 3 warm-up epochs, for a total of 15 epochs. Similarly, the best model on the validation set was selected. Gradient accumulation yielded an effective batch size of 150. Clips were letterbox-padded and normalized; augmentations included random horizontal flipping and temporal jitter of the bounding boxes to simulate detector noise. To further mitigate imperfect bounding boxes, we added 50 pixels of padding to each side of the bounding box to make sure the entire animal was visible.

The ClinObs classifier is an extended version of the behavior classifier. Consistent with the approach used for the behavior model, this expanded dataset also included examples designed to represent nuanced clinical observations and the challenges in distinguishing them from similar behaviors and background activity (Table 1).

Building upon the behavior classifier, we trained the ClinObs model using the same ViT-S/16 architecture with joint space-time attention (Dosovitskiy et al., 2020; Arnab et al., 2021; Tong et al., 2022), initialized from a VideoMAE checkpoint pretrained on Kinetics-400 (Tong et al., 2022). We split the data into training, validation, and test sets using the same 60%-15%–25% ratio, still maintaining balanced representation of day/night conditions and bars/glass environments. We used a batch size of 14 with gradient accumulation over 10 batches, yielding an effective batch size of 140. The model’s input was the same as the behavior model: 224 × 224 images, with each input clip consisting of 16 frames sampled with a stride of 4. We still address the class imbalance with random weighted sampling. We trained the model for 75 epochs, including a 15-epoch warm-up period, utilizing an AdamW optimizer with a cosine decay learning rate schedule (base 1 × 10⁻³, minimum 1 × 10⁻⁶ and weight decay of 0.05). Once more, the best model on the validation set was selected. Consistent with the behavior model, we applied letterbox padding and normalization of the clips, implemented temporal jitter augmentation, and added 50 pixels of padding to each side of the bounding box. To prevent overfitting, we implemented mixup (Zhang et al., 2017) and cutmix (Yun et al., 2019) augmentation techniques, alongside RandAug (Cubuk et al., 2020) with 9 augmentations and a magnitude of 15. As before, we split eating and eating-bowl during training; additionally, we split ataxia and IVM into subcategories representing varying degrees of severity (slight-moderate-severe) during training, with these subcategories merged for testing.

We selected six female Beagle dogs from our colony to wear the three different harness-types: Y-St (two animals), B-Sq (two animals) and R-N (two animals). Animals were group-housed with three groups of N = 2 in a room with bars to include all three possible harness combinations. Subsequently, groups were reorganized into two groups of N = 3 in a room with glass fronts, followed by bars. After a habituation period in a room with bars, all animals (two groups of N = 3) were equipped with an Actiwatch-Nano® accelerometer attached to their harnesses to record their activity.

For the activity tracking validation, 2-h videos were selected with varying activity levels in all animals, both during day and night (Table 2). The Actiwatch-Nano® provided one read out (activity count) per minute. These were compared to the AI activity levels which were calculated using the Euclidean distance (in pixel, as explained above in “AI activity tracking and mobile/immobile” and Figure 4A). Accelerometer and AI data were compared in two ways: i) by visual inspection of individual actigrams; and ii) by assessing their correlation. For the latter, we calculated the non-parametric Spearman correlation coefficient using Graph Pad Prism 10.1.2 for activity read-outs per minute, and binned per 15 min and per 2-h period.

To validate our tracking and reID model, snippets were specifically selected to include a variety of interactions in group-housed animals to challenge our model: walking or running around, animals crossing each other, playing, resting separately and together, etc. (Table 2). The ground truth ID was manually established on every frame and tracking metrics were calculated using the motmetrics python library (Heindl, 2019).

For the validation of the behavioral classifier, we evaluated the performance of drinking, eating from a food hopper, eating from a bowl, and none (Table 2). The none category was evaluated on the eating and drinking fragments as they contained typical none behavior like animals walking around, jumping on and off bench, animal-animal interactions, etc. To challenge our model, we also included several not eating and not drinking fragments. The overall confusion matrix was generated by summing frames across all videos for both the manually established ground truth and AI model.

In a final real-life validation, 4-h footage spanning the entire food access period, was analyzed for 10 animals (Table 2). Our model predictions were manually checked on video and the entire footage was also visually checked for missed detections by the model. Accuracy was calculated on event-level: #   e v e n t s   c o r r e c t l y   d e t e c t e d t o t a l   #   p r e d i c t e d   e v e n t s + m i s s e d   e v e n t s   x   100

The validation of mobile/immobile was done in two parts (Table 2).

For the first part, 10 min fragments were selected of single-housed animals from prior studies in which either increased or decreased activity was registered. The ground truth whether the animal was mobile or immobile was manually established on every frame. To determine the most optimal cutoff for mobile/immobile classification, an ROC (Receiver Operating Characteristic) curve was calculated that plots the TPR (True Positive Rate) in function of the FPR (False Positive Rate) for every cutoff. We evaluated values between 0 and 20 with 0.01 increments and calculated the Youden’s Index for every point to find the best trade-off between TPR and FPR: Y o u d e n ′ s   I n d e x = s e n s i t i v i t y + s p e c i f i c i t y − 1 = T P R − F P R

The highest Youden’s Index of 0.91 corresponded to a cutoff of 10.0. This cutoff was subsequently validated in group-housed animals.

The same 10 min video fragments with increased/decreased activity were analyzed to have a more balanced distribution of the more “excited” poses (standing, standing up and standing down) vs. the more ‘quiet’ poses (sitting and lying) (Table 2).

We selected nine female Beagle dogs and randomly divided them into three groups of three animals wearing the different harness types. All animals were observed for two baseline days prior to dosing. On the third day, each group received a different treatment by oral gavage: 1) Acepromazine 2.5 mg eq/kg (Sigma-Aldrich®, Burlington, MA, US); 2) Transcutol® HP 1500 mg/kg (Gattefossé, Saint-Priest, France); and 3) Water as vehicle control (Figure 7a). As per standard protocol, we observed the animals in person for any change in behavior and/or ClinObs during specific time periods post-dosing: 5 min, 30 min, 1 h, 2 h, 4 h, 7 h, and 10 h. To assess whether our AI model was able to detect the same ClinObs and behavioral changes as noted during the in person observations, we analyzed the 24-h footage of both baseline days and the dosing day using our AI pipeline, followed by human QC of significant/aberrant AI signals (as illustrated in Figures 7c,d). For visualization purposes, framewise AI predictions were summarized per minute by binning the number of predictions per class within that minute.

On top of the framewise predictions, we implemented an objective, adaptive baseline thresholding strategy to highlight significant differences between the baseline and treatment days. For this, a moving average with a 5-min time window was applied on the framewise predictions to account for the temporal aspect. Individual thresholds were then calculated for each animal and each ClinObs class by determining the minimum number of predictions per minute required to suppress all predictions for that class during the baseline period. This threshold was subsequently applied to the dosing day data.

Out of the several detector ANNs evaluated (Supplementary Figure S1A), the best performing model was YOLOv2 (Redmon and Farhadi, 2017). Our detector ANN reached both high precision and recall, reflected by a F1 score of 94.8% (Figure 3a). Performance was slightly better on daytime footage (F1 of 96.1%) than during night (91.1%). This was expected given that approximately 65% of our data contained day footage, while nocturnal conditions pose greater challenges. To verify the accuracy of the generated bounding boxes that downstream ANNs rely on, we calculated the average Intersection over Union (IoU) between the detector output and the annotated bounding boxes. Our detector ANN reached an overall average IoU of 80.5%, with a lower IoU at night (73.9%) compared to day (82.7%) due to reduced visibility in greyscale imaging and the natural tendency of canines to cluster together while sleeping. These values are considered very high since an IoU ≥50% is typically regarded as a true positive match in computer vision literature (Everingham et al., 2010; Redmon and Farhadi, 2017).

A TIDE (Toolkit for Identifying Detection and segmentation Errors) analysis (Bolya et al., 2020) offered insight in the sources of errors our detector model made. It showed that all types of mistakes made by our detector ANN occurred more during the night (≤3%) compared to the day (≤1%), correlating to the ANN’s decreased performance (Figure 3a). Most errors at night were missed detections followed by duplicate detections (canines were generally not wearing harnesses in this footage) and localization errors (low IoU overlap). The detector ANN showed very low background detections (erroneous identification of non-existing canines) across both conditions, indicating a low propensity for false positives.

Note that YOLOv2 was relevant at the time of our research. Although, in the meantime novel architectures are available, we opted not to test them as our detector model and the resulting animal tracking (Section 3.3.3, 3.4.2.) are very performant and the potential gain would be minimal.

Before collecting a large dataset to train a robust animal identification model, we initially gathered smaller datasets to determine the most effective approach (see Methods 2.4.2, Supplementary Figure S1B). This preliminary step allowed us to evaluate different strategies and select the best-performing methods for our final model.

Our final approach utilizes an ResNet-18-based ANN that can classify 3 IDs based on color during the day and the reflective pattern during the night: Black-Reflective squares (B-Sq), Red-No Reflection (R-N), and Yellow-Reflective stripes (Y-St) (Figure 1b). This ANN was trained on a larger dataset that included more challenging data, such as occluded animals and image cut-outs that contained multiple harnesses (Table 1). Our final harness classifier ANN achieved an overall top-1 accuracy of 80.3% with recognition of the different harness types varying from 71.9% (B-Sq) to 85.0% (R-N) (Figure 3B).

Our strategy of combining a modified Jonker Volgenant algorithm (Jonker and Volgenant, 1987; SciPy, 2008; Crouse, 2016) with a Kalman filter across multiple camera views (Figure 2) resulted in excellent tracking and reID performance (Figures 3c–f), where reID (re-identification) is defined as assigning the correct harness color (ID) to every animal on every frame. In general, there were no notable differences in tracking performance between separate and clustered animals, apart from a tendency of more fragmentations in the clustered videos. This was also the case for day and night, where day scored only marginally higher for IDF1, IDP and IDR. Across all conditions, we reached average scores of 85.1% MOTA, 93.0% IDF1, 98.4% IDP and 88.4% IDR; and 11.0% missed detections (Figures 3c–f).

IDF1 scores for both separate and clustered animals were outstanding for day (∼94%) and night (∼91–92%) conditions, implicating that the correct ID was assigned to nearly each detected animal (Figure 3c). This was also reflected in the close to perfect IDP score (Figure 3f). Of the 71 ground truth identities, 57 IDs were mostly tracked (tracked for more than 80% of their lifespan), 11 were partially tracked (between 20%–80%), and only 3 were mostly lost (less than 20%) (Figure 3e). The MOTA score accounts for missed detections, false positives, and ID switches, which typically leads to lower values than IDF1 that focuses more on identity preservation. Nevertheless, our MOTA scores of 77.0%–89.5% still indicated strong tracking performance (Figure 3d). Indeed, apart from a few outliers, the number of missed detections was low across all video snippets, which is also mirrored in high IDR scores (∼86–90%) (Figure 3f). Considering the huge number of 23,283 targets (i.e., individual animal cut-outs), we observed a negligible number of switches, fragmentations and false positives (Figure 3f).

The fact that our IDR score was higher than the performance of our harness classifier model alone, implied that the Jonker-Volgenant algorithm had a positive effect on ID consistency.

The pose classifier receives cut-outs from a single canine through time and predicts the pose on a frame level (Figure 1a). Our ResNet-18 model reached a top-1 accuracy of 95.2% when using detector-generated cut-outs instead of manually annotated ones (Figure 3d). Lying and Standing classes were the most performant (accuracies of ∼97%), followed by Standing Up (88.7%) and Sitting (82.9%). Standing Down was the least performant class with 55.0% accuracy (Figure 3d), also related to the very low number of occurrences in our dataset compared to the other poses (Table 1).

Our initial ViT Small model for behavior was trained for drinking and eating (Figure 3e). The model achieved a top-1 accuracy of 94.0% and a class accuracy of 92.7% calculated on a frame-level. It performed well on both behavior classes, with accuracies of 93.1% for drinking and 89.6% for eating, while rarely confusing these (Figure 3e). When errors occurred, they were typically misclassifications between a behavior and the background class (none), or vice versa, suggesting that the model has learned to distinguish these behavioral patterns. Furthermore, high accuracy on the background class none (95.4%) indicates a low number of false positives (Figure 3e).

Building onwards from the ViT Small Behavior model, our ClinObs model demonstrated promising results in detecting 11 distinct behaviors and clinical observations on a frame-level, achieving a top-1 accuracy of 79.0% and a class accuracy of 47.9%. Our model maintained strong performance on behavioral classes such as drinking and eating, highlighting its ability to reliably detect common observations. As expected, classes with limited representation in the training data such as anxiety, limb stiffness, and vomiting presented greater challenges for the model. Because of this extreme—but unavoidable—training data’s class imbalance, the performance ranged substantially between classes: drinking (93.2%), none (90.8%), eating (87.3%), circling (86.6%), convulsion (43.2%), ataxia (40.4%), head shaking (40.1%), limping (35.2%), involuntary muscle movements (IVM, 30.1%), anxiety (20.0%), vomiting (6.1%) and limb stiff (2.1%) (Figure 3f). Notably, our model occasionally confused IVM with convulsion (4.9%), convulsion with ataxia (16.6%), and limb stiff with limping (30.4%), which is understandable due to the visual similarity of these ClinObs (Figure 3f). Because of the inherent class imbalance, the model showed a greater inclination toward predicting the background class None. Across nearly all classes, the precision was higher than the recall, indicating that the model leans more towards correctly predicting instances, at the expense of missing some true positive events.

Accurate AI Activity tracking relies on correct outputs from the detector, harness classifier and tracker (Figure 1a). To validate our AI Activity tracking, we compared it to Actiwatch-Nano®, an accelerometer-based device routinely used during nonclinical studies performed in this facility. We calculated the animals’ activity based on the movement of their bounding boxes (Methods 2.4.4, Figure 4a).

After normalization of both different metrics, 2-h actigrams for both activity measurements were visually very similar for all animals, across a range of activity levels (Figures 4b, 5a). During daytime, normalized accelerometer values were sometimes “larger” compared to AI. These generally corresponded to periods of excitement in which animals were jumping or rubbing on the floor. As the accelerometer incorporates movements over x, y and z-axes, these short, intense movements resulted in very high counts while the actual “distance moved” was much smaller. As our AI activity tracking uses a uniform pixels-based measurement irrespective of the intensity, it resulted in a more objective representation of the actual “distance moved”. For linear movements (animals walking or running), there was a perfect overlay of activity read-outs of both techniques upon normalization. During the night, AI activity patterns were still highly similar to the accelerometer but showed more differences due to imperfect bounding boxes (night is a more challenging scenario, see also Figure 3a) and occasional ID switches. Similar to daytime, AI activity tracking more accurately represented the ‘distance moved'.

The similarity in the two methods was emphasized by a strong correlation between AI and accelerometer activity outcomes across all time spans: r = 0.8825 for 1-min intervals which improved further to 0.9257 and 0.9650 for 15 min and 2 h respectively (Figure 4b).

Along with detailed actigrams, it is useful to have a more general overview of the animals’ activity by classifying the time spent as either mobile (moving from one point in space to another) or immobile (remaining in the same location). In single-housed animals, our AI model achieved an excellent classification of both mobile (96.8%) and immobile (94.0%) occurrences (Figure 4c, left panel). Most errors occurred in immobile classifications, due to imperfect bounding boxes or bounding boxes changing shape without animals actually moving, e.g., when tail wagging or heads moving while remaining in the same location. Equally, in group-housed animals (N = 3), we reached a very good classification accuracy despite this challenging setting: 90.4% for immobile and 81.0% for mobile (Figure 4c, left panel). The drop in performance was primarily due to imperfect bounding boxes when animals cross each other and/or at night.

While the impressive results described above already imply strong reID and tracking, we manually assessed their performance in detail on several 5-min snippets as described in the Methods section and illustrated for bars setup in Figure 5a. For the latter during daytime, all tracking metrics were close to perfect, reflected in 32 of 33 total IDs being mostly tracked (>80% of their lifespan). At night, there were more missed detections due to the challenges of recognizing animals in greyscale colors and their tendency to cluster together while sleeping. This is reflected in lower MOTA and IDR scores, and 16 out of 30 IDs being partially tracked. However, IDP scores remained high, indicating that when an animal was detected, the correct ID was assigned most of the time (Figure 5b).

The small differences in reID performance between the two evaluated groups (N = 3 each) are attributed to chance and individual behavior differences, e.g., whether they like being close to each other and their preference where to stand, sit or rest. Merging results for both groups in the bars setup across day and night yielded an excellent overall reID accuracy of 95.3%–96.2% and average scores of 85.1% MOTA, 91.4% IDF1, 100% IDP, 85.3% IDR and 14.9% missed detections (Figure 5c).

For the setup with glass fronts and group sizes of N = 3, we achieved a similar outstanding reID accuracy ranging from 92.5% to 100% across day and night footage. Also tracking metrics were similar, with high IDF1 (87.0%) and IDP (100%) scores; and occasional missed detections (21.0%) which are reflected in the MOTA (79.0%) and IDR (79.9%) scores (Figure 5c).

In the assessment of reID and tracking performance for three harness combinations with group sizes of N = 2, all combinations achieved at least 94.8% reID accuracy across day and night. The R-N/Y-St combination proved to be superior with the fewest missed detections (5.1%) and highest scores across all metrics (94.9% MOTA, 97.2% IDF1, 100% IDP and 94.9% IDR) (Figure 5c).

Our pose classifier achieved an excellent recognition of Standing (97.4%) and Standing Up (94.6%) when implemented in the pipeline, followed by Standing Down (86.1%). Lying and Sitting classes were the least performant with accuracies of 70.4% and 66.5% respectively. Lying was quite often confused with Standing, and Sitting was mixed up with Lying and Standing (Figure 4c, right panel).

Our behavioral classifier demonstrated excellent frame-level performance when implemented in the pipeline (Figure 6b), similar to the test set (Figure 3e). For eating, we assessed four different conditions (Figure 6d) where Eating hopper-glass consistently excelled across all fragments, followed by Eating Bowl-Bars (Figure 6a). The Hopper-Bars and Bowl-Glass conditions showed more variation between different fragments. However, the lowest-performing video still achieved 73% frame-level accuracy in both conditions. Overall, eating reached an outstanding frame-level accuracy of 93% (Figures 6a,b).

Drinking was in general excellently detected for both glass and bars during day and night, with at least 84% accuracy and often reaching 100% (frame-level). Only two snippets showed low accuracy of 47% and 3% (Figure 6a) due to the animal pausing frequently when drinking and an inadequately predicted bounding box that did not include the animal’s head, respectively. Across all snippets, drinking showed an excellent frame-level accuracy of 90% (Figures 6a–d).

Finally, the background class none achieved a frame-level accuracy of 99% (Figure 6b), with incorporation of the challenging snippets comparable to the two behaviors (not eating and not drinking), again indicating that our model generates a low number of false positives. Similar to our test set, the model never mixed-up eating and drinking; misclassifications were limited to confusion between the behavior and the background class (none). Figure 6c also illustrates the near-perfect overlay of our AI predictions on the manual ground truth at frame level.

In real-life situations, accurate frame-by-frame predictions are less critical as long as overall events are detected. Analysis of continuous 4 h food-access footage from ten animals with different eating patterns (2 representative examples shown in Figure 6e) revealed high event-level accuracy of 96% for eating and 91% for drinking, with minimal missed or incorrect detections.

We performed a validation study with two reference compounds (Acepromazine 2.5 mg eq./kg P.O. and Transcutol® HP 1500 mg/kg P.O.) and one control group (water) to validate our AI pipeline’s ability to detect clinical signs in a realistic experimental setting (Figure 7a). For this, we investigated whether our model could identify the same behavioral changes and ClinObs as noted during the standard in person observations.

Transcutol® HP is a commercially available vehicle that induces ataxia and IVM (in canines), when used in its pure form (internal data). Our model accurately detected ataxia in all animals within the same period as the in person observations. In one animal, limping was also occasionally assigned together with ataxia (Figure 7c). When implementing our adaptive baseline thresholding strategy to filter out baseline noise and highlight significant changes on the dosing day, clear Ataxia signals remain within the expected time window (Figure 7e). In addition, our model detected IVM in the same two animals as the in person observations with a slight difference in IVM duration (Figure 7c).

Acepromazine is a known sedative that induces unstable gait (ataxia-like) and IVM (in canines)(VCA-Animal-Hospitals, 2023). Our AI activity tracking successfully detected the decreased activity observed during in person assessments in these animals compared to baseline (Figure 7b). Despite the sedative effect with only short periods of movement, our model successfully identified the ataxia-like gait as either Ataxia or Limping in all animals, during the majority of the expected time frame (Figure 7d). As they are both phenotypes of abnormal gait, AI outputs for Ataxia and Limping signals were merged together for the manual analysis to boost the model’s performance and enable better differentiation compared to the baseline noise. When plotted together with the animals’ activity, true positive signals (high abnormal gait-to-low activity) could be clearly distinguished from the baseline noise (low abnormal gait-to-high activity) (Supplementary Figure S3). Furthermore, our model successfully detected the IVM in all animals within the expected time window with exception of one animal where the AI detections were shorter in duration compared to in person observations (Figure 7d). Upon implementing our adaptive baseline thresholding strategy, clear IVM signals remain, although fewer in number as (also correct) baseline events are filtered out (Figure 7e).

A control group was included to investigate whether this mock-treatment would result in observable differences in AI signals after dosing compared to baseline. While our model correctly detected IVM and ataxia-like signals in some animals on the treatment day, the AI signal patterns were highly similar compared to baseline (Supplementary Figure S2A,B). Similarly, applying our thresholding strategy resulted in almost no notable signals (Supplementary Figure S2C,D), again showing that our model can identify actual treatment-related observations.

As final part of the field validation, we performed a manual video analysis for a large number of substantial AI signals and/or patterns to evaluate the precision of our model detections on event-level (Figure 7g), opposed to the framewise precision on the test set (Figure 3f). As mentioned above for the Acepromazine-dosed group, we pooled ataxia and limping detections for this analysis as an abnormal gait phenotype. For almost all model detections in the study, we showed strong to excellent precisions: circling (70%), IVM (70%), ataxia-limping (72%) and head shaking (95%). While convulsion detections showed 0% precisions, it is important to note that in many detections, the animal was rubbing (35%) or showing ataxia (6%); which can be highly comparable to an actual convulsion.