In order to assess fisheries stock for a target species, it is necessary to estimate its abundance and biomass spatially and across time, but also along the species length structure (Gulland, 1983). Such information may be insufficient or biased when acquired from landings of data-poor fisheries, thus calling for independent methods to complement traditional fisheries stock assessments (Moore et al., 2013). The emergence of video-assisted methods like BRUVS (Baited Remote Underwater Video Stations) (Whitmarsh et al., 2017) using low-cost small action cameras may provide such valuable complementary information (Moore et al., 2013; Letessier et al., 2015). However, video-based assessments require a considerable processing time to manually count fish on images, limiting their broad-scale applications (Sheaves et al., 2020). Modern automated video analyses using deep learning algorithms are becoming more accurate (Villon et al., 2018; Marrable et al., 2022; Bhalla et al., 2024) and may reduce these costly video-processing constraints (Tseng and Kuo, 2020; Connolly et al., 2021; Lopez-Marcano et al., 2021). Yet the lack of labelled, species-rich, datasets for fish classification and identification keeps its automation binding. Furthermore, the performance of deep learning algorithms on deep video surveys, including darkness, artificial lightning or generally variable image conditions and backgrounds, is still poorly known (Saleh et al., 2024; Jian et al., 2024).

In the Pacific, deep-water demersal fisheries are of high significance not only for local consumption but also for their commercial, cultural, and recreational value (Dalzell and Preston, 1992). Their commercial development began in the 1970s to alleviate fishing pressure on coral reefs but has generally collapsed in the 1990s (Williams et al., 2012). Over time, these fisheries have transitioned primarily to subsistence but continued to hold commercial significance in more developed and isolated regions like New Caledonia and Hawaii (Newman et al., 2016). While deep demersal fisheries include around 200 species in the western Pacific Ocean, the landed species are mainly composed of snappers, a group in the Lutjanidae family associated to the genera Etelis, Pristipomoides, Aphareus, and Aprion. Deep-water snappers are characterized by relatively slow metabolic rates and long lifespans, making them highly vulnerable to overfishing (Newman et al., 2016; Wakefield et al., 2020). Usually found at depths starting at 100 m to 500 m and more, these fish aggregate in structured topographies like steep slopes, seamounts, or any topographic anomaly such as sand banks or pinnacles (Gomez et al., 2015). Yet, deep-water snapper fisheries lack core management measures based on stock assessments which remain challenging to perform in such hardly accessible marine habitats (Newman et al., 2016).

Baited Remote Underwater Video Stations are among the most used, standardized video technics to study underwater fish ecology (Whitmarsh et al., 2017; Langlois et al., 2020). BRUVS can assess spatial and temporal variation in fish assemblages through visual identification and quantifying species abundance (Letessier et al., 2015; Wellington et al., 2018). They are a low-cost method able to generate large amounts of data (Cappo et al., 2007; Osgood et al., 2019; MacNeil et al., 2020). BRUVS can be deployed in a variety of habitats, including coral reefs, but also soft sediments, freshwater, the deep sea, or the pelagic environment (Ellender et al., 2012; Gladstone et al., 2012; Zintzen et al., 2012; Henderson et al., 2017; Schmid et al., 2017; Letessier et al., 2019; Reis-Filho et al., 2019). Their use in environmental monitoring is increasing with more studies focusing on industrial settings like underwater pipelines (Bond et al., 2018; Schramm et al., 2020, 2021) or windfarms (Griffin et al., 2016). BRUVS are also emerging as independent and complementary methods for fisheries stock assessments (Cappo et al., 2004; Ault et al., 2018; Boldt et al., 2018). Clearly, BRUVS show great potential for monitoring deep-sea fisheries.

When manually processing BRUVS footage by identifying, counting, and measuring fish, the fastest and commonly used metric is the MaxN (Whitmarsh et al., 2017; Langlois et al., 2020). MaxN corresponds to the maximum number of individuals per species that can be counted in a single image per video. While conservative, this measure prevents from counting the same individuals twice. It has been shown that getting accurate fish abundance measures on each image from a video station, or within short video periods, and averaging these measures along the whole video may be more representative, but would multiply processing costs (Schobernd et al., 2014). This cost could effectively be reduced using deep learning algorithms.

Deep learning and specifically Convolutional Neural Networks (CNNs) are artificial intelligence algorithms that generate classification by autonomously identifying features in images (LeCun et al., 2015). The rapid progress in the automatic processing of underwater images has already long permeated in ecology with the accurate detection of several marine species (Christin et al., 2019; Mannocci et al., 2021; Saleh et al., 2022; Xu et al., 2023). The ability to detect and identify fish on images in their natural environment has also been explored, but have mainly targeted coral reef fish, which can be highly differentiated due to their diversity of shapes and colors (Mandal et al., 2018; Villon et al., 2018, 2022; Saleh et al., 2024). The available public images follow the same trend but are diversifying on shallow habitats, with images from fish at deeper strata still lacking (Saleh et al., 2024; Bhalla et al., 2024).To our knowledge, few studies have used deep-water images with their own singular constraints like variable light levels (Saleh et al., 2022; Jian et al., 2024 but see Liu et al., 2023), and none for the deep-water snappers. Given the diversity of habitats and conditions in which fish can be detected, incorporating more diverse species and backgrounds is crucial for improving general fish detection and identification techniques (Saleh et al., 2022; Bhalla et al., 2024; Jian et al., 2024).

The state-of-the-art of object detection and classification features three primary algorithms: Single Shot Detection (SSD), Faster Region-based Convolutional Neural Network (Faster R-CNN), and You Only Look Once (YOLO) (Bhalla et al., 2024). While YOLO and SSD have demonstrated notable speed advantages over Faster R-CNN, the latter has shown superior accuracy in object detection and classification (Kim et al., 2018; Bose and Kumar, 2020; Kaarmukilan et al., 2020; Lee and Kim, 2020; Lee et al., 2021; Mahendrakar et al., 2022; Sarma et al., 2024). This difference is due to the NAS (Neural Architecture Search) automatically searching and building the most efficient architecture (Elsken et al., 2018). Furthermore, while some recent versions of YOLO do outperform older Faster R-CNN implementations, one of YOLO’s weaknesses is its inability to address important variation of object sizes like Faster R-CNN can do (Ammar et al., 2019). Such variation is commonplace in underwater videos, where individuals can appear either close or very far from the camera. One of the main advantages of YOLO is its speed in real-time detection operations where faster R-CNN will take more processing time. BRUVS are usually deployed and retrieved over a short period of time, leading to an inevitable separated processing time from deployment. For this reason, Faster R-CNN seems to represent the best option for this context as being the most precise although a little bit slower (Sarma et al., 2024).

Here, we chose the Faster R-CNN architecture and assessed its ability to automatically detect deep-water snapper species in BRUVS images from deep slopes and seamounts of a South Pacific island: New Caledonia. We then discuss constraints and solutions about how this algorithm may help accelerate video processing for fisheries stock assessments considering a fully automatic and semi-automatic approach. To our knowledge, this study is the first to train an artificial intelligence algorithm for the detection, identification and counting of deep-water snapper in the wild on baited videos.

The main contributions of this article are as follows:

Faster R-CNN training lasted for four days in order to execute 200,000 iterations on the multi-GPU calculator. Out of the 1,333 testing images comprising 2,318 annotated fish, the trained Faster R-CNN automatically detected 2,351 fish, out of which 1786 were true positives (76%) so 565 were false positives.

The F-measure of automatic detections ranged between 0.15 to 0.87, indicating considerable variation in the evaluation measures per species. Largest values were obtained for Etelis coruscans (F-measure: 0.87, recall: 0.91, precision: 0.84), closely followed by Pristipomoides filamentosus (F-measure: 0.79, recall: 0.86, precision: 0.73, Table 2 ). Pristipomoides multidens, was not detected on any of the 13 testing observations, hence values of 0 for the recall and precision. Pristipomoides zonatus was hardly detected in the equally low testing observations (recall of 0.08 on 12 annotations). However, the model never classified another deep-water snapper as this species (precision of 1.0). These two latter species along with Etelis carbunculus and Parapristipomoides squamimaxillaris, were those with less than 89 annotations to train the model. Species with comparatively higher annotation numbers (> 186, Aprion virescens, up to 5729, P.filamentosus) showed F-measures of at least 0.71 (Pristipomoides flavipinnis). A sample of the testing dataset is illustrated in Figure 1 .

The semi-automatic approach, in which the expert corrected classification errors, showed an F-measure ranging from 0.15 to 1 ( Table 2 ). A drastic increase in performance metrics was observed for species with higher number of annotations (>186), with semi-automatic F-measures ranging from 0.86 for A. virescens to 1 for E. coruscans, which showed no more false negatives in the testing dataset. P. filamentosus, with the highest number of images tested (1,303), returned an F-value of 0.97 compared to 0.86 without correction. The largest increase in F value was for A. rutilans and P. flavipinnis, from 0.66 and 0.65 to 0.86 and 0.89, respectively.

For the analysis of fish abundance on whole BRUVS (MaxN), we focused on species with F-measures superior or equal to 0.71 (aka with more than 100 annotations, Table 2 ) as models with lower F-measures provided poor abundance estimates. High correlation coefficients were observed between manually and automatically estimated fish abundance ( Figure 2 ; Table 3 ). Pearson Correlation coefficient ranged between 0.72 and 0.90 among species, with the highest values observed for Etelis coruscans and an overall value of 0.85 when combining data from all species.

The slope coefficient for each and all individual species were significantly different from zero (p< 0.001). However, while automatic fish abundances appeared comparable to manual abundances for up to three to four individuals in the same frame, the Faster R-CNN model tended to underestimate higher abundance with slope coefficients significantly smaller than 1 for all and each species. Slope coefficient ranged nonetheless between 0.65 and 0.88 with highest value found when considering all species together ( Figure 2 ). Except for P. argyrogrammicus, all intercepts were significantly different from zero, but with marginal deviation (range: 0.01 to 0.05 except for P. filamentosus: 0.31).

The semi-automatic protocol yielded fish abundance estimates much closer to manual counts, with a Pearson correlation coefficient of 0.96 for all species combined ( Figure 2 ; Table 3 ). Correlations ranged from 0.86 to 1 depending on the species. E. coruscans showed a perfect fit (slope = 1, intercept = 0) with semi-automatic MaxN identical to manual MaxN. The slope between MaxN_Man and MaxN_Semi was not significantly different from one for P. filamentosus, revealing extremely good semi-automatic model performance.

The use of the Faster R-CNN algorithm to automatically detect, identify and count deep-water snappers proved successful and highly promising considering the challenge this group of fish presents and the variable background habitat. The algorithm effectively differentiated between species that were very similar and hard to distinguish, even for an experienced taxonomist. While the detection and identification will probably need post-verification until enough annotations are gathered to achieve automatic F-measures above 0.9 for all species, the abundance estimations were still consistent with manual counts. This procedure can already be employed for automatic deep-sea snapper monitoring, or semi-automatic monitoring, where observers would save substantial processing time by simply verifying and adjusting detections rather than processing entire BRUVS videos.

It is crucial for fisheries stock management to be able to work on the species level. This deep-water snappers’ dataset represents a fine addition to the collection with varying habitat constraints such as presence or absence of natural light and hard and soft substrates. Especially, this species group is challenging due to the similar appearance of its members. Deep-water snappers are mostly “greyish”, “fish-looking” species, posing a challenge in identification, particularly for P. filamentosus and P. flavipinnis which share almost identical characteristics ( Figure 3 ). E. coruscans stands out with its reddish color and long elongated tail tips, allowing the algorithm to distinguish it from other species, and lead to the highest recall, precision, and F-measure metrics. Furthermore, the semi-automatic treatment of E. coruscans yielded individual detections and abundance values that matched the manual estimates perfectly. This is highly encouraging, considering E. coruscans is a highly targeted species of this fishery (Newman et al., 2016). P. filamentosus had the highest number of annotations and images, which likely explains its high identification success rate. The bigger the training database per feature, the better the identification for the Faster R-CNN algorithm, which typically requires at least 1,300 training images per feature to achieve over 95% certainty in fish identification (Villon et al., 2018). In our study, only two out of the 11 species studied (P. filamentosus and P. flavipinnis) met this training size requirement.

While the human observer may browse through the video sequence to observe color, behavior, movements, and other clues to identify species and count individuals, the algorithm is restricted to each single image to decide. That the algorithm was able to effectively differentiate between snapper species with so little information at hand is therefore very encouraging. However, errors have still been observed with many false positives caused by rarer species (e.g., A. rutilans) being confused with the most common ones (P. filamentosus, Figure 1 ). This type of confusion was easily corrected by the intervention of an expert during the semi-automatic counting protocol as the fish still got detected. The expert fully corrected each false positive, and the precision became equal to one. Additionally, semi-automatic recall also increased compared to its value with the automatic protocol. This is because some fish were not detected in frames where other individuals were detected. Since the expert corrected the entire frames, undetected individuals were also annotated, thereby reducing the number of false negatives and increasing recall. For example, the recall of A. rutilans increased from 0.66 to 0.80, indicating that this species was present in many frames with other detected species. However, the recall of A. virescens remained the same, indicating that no further detections of this species occurred on frames where other snappers were detected by the algorithm. The confusion problem between species could be partly due to the disparity in available images between similar species, with those with fewer images being misclassified more often than those with more images. While a semi-automatic protocol can partly address the issue, an alternative solution might involve adding temporal information through motion analysis or a tracking algorithm that would isolate the background or follow the same individuals, thereby adding detection and identification information from previous frames to subsequent ones (Shin, 2016; Jalal et al., 2020). The other major constraint highlighted in this study is the underestimation bias at higher abundances. We observed that frames involving many fish can become easily saturated (notably P. filamentosus, cf. Figure 1A ) with few individuals blocking the camera’s field of view. This bias in the algorithm seems rather inevitable due to its dependance to the technical video sampling system with a single sensor and angle of view. The MaxN abundance index based on the maximum number of fish present in the same frame is known to be sensitive to the phenomenon of image saturation (MacNeil et al., 2020). It is also reported in another study working on a different species of snapper in a different configuration (daylight reef) (Connolly et al., 2021). Our semi-automatic protocol could correct this bias for the two species that presented the highest MaxN, E. coruscans and P. filamentosus, yielding F-measure > 0.96 after correction by an expert taxonomist. The tracking of individuals across successive frames might also permit a better differentiation of individuals saturating images, hence reducing or removing the bias in MaxN at high abundance, as the expert usually also does.

We are confident that our trained Faster R-CNN algorithm is already operational for fisheries assessment using our semi-automatic detection procedure. The whole process using BRUVS to assess fish abundance is nondestructive, independent from fisheries data and may today become cost-effective with the support of artificial intelligence. Our model, as it is, can provide a matrix of detections per species for each frame of the video stations. The frames with the greatest number of detections per species can then be identified and used as references to define video intervals of a few seconds including the MaxN of the different species. These short video sequences could then be processed by biologists using programs like EventMeasure, reducing hours of video processing to minutes. Furthermore, the manual processing of the short video sequences would be limited to simply correcting algorithmic detections, which would further speed up the process. Additionally, new annotations should be used to retrain the algorithm and further improve its performance. If stereo cameras are used, then fish size could be measured in addition to abundance. Although size-measurements are performed manually so far using programs like EventMeasure (Letessier et al., 2015), algorithms exist to automatically measure object dimensions on videos like with instance segmentation (Othman et al., 2018; Garcia-d’Urso et al., 2022). Their ongoing development represents the next stage and their application on BRUVS and fisheries management is warranted.

Some caveats can still be discussed for further improvement. The uneven distribution of training images among species calls for an increased sampling to complete the dataset and improve identification accuracy (Villon et al., 2018). Our current algorithm may still drastically reduce annotation times for rarer species as they are detected but mostly confused with more occurring species. However, rarity is a key characteristic of biodiversity, and a large number of annotations can remain difficult to gather for the rarest species (Villon et al., 2022). In this case, methods like the few shots deep learning algorithm could be coupled with the Faster R-CNN to compensate for the lack of annotations (Villon et al., 2021). A coupling with other BRUVS datasets from other regions may also improve the algorithm performances but may then face issues related to changes in environmental conditions across regions (Kalogeiton et al., 2016). However, while our study relied on a dataset restricted to New Caledonia, the sampling occurred across the spatially immense EEZ and across depth ranging from shallow photic seamounts (50-60 meters deep) to deep aphotic seamounts and continental deep slopes (150-500 meters deep), exploring diverse environmental backgrounds and light intensities (Baletaud et al., 2023).

While this case study involved a particularly constraining group of species (looking-alike deep-water snappers), in variable background conditions of light and habitats, it further shows that the Faster R-CNN is a worthy algorithm architecture that may be used in many use-case scenarios involving fish species detection. The methodology is applicable to any visually identifiable fish species provided sufficient training images for the model, as is the main constraint for any deep learning development (Ahmad et al., 2023). New CNN architectures are released more and more frequently, improving classification speed and accuracy, and their review using this new dataset will prove interesting although not in the scope of this study. The potential for deep learning to improve the day-to-day work of marine scientists in monitoring fisheries seems certified for the future (Zhang et al., 2021). The transition is progressive, and a semi-automatic approach may be yet closer to being adopted by operational monitoring organizations or consultancy firms using this work.