The ROV used for this study (Boxfish™ ROV, Boxfish Research Ltd) is relatively small (length 60 cm, height 30 cm, and width 40 cm), and is equipped with high-powered LED lights, a Sony™ RX100 (V) camera capable of recording in 4K (30 fps) and four parallel scaling lasers. The ROV is relatively lightweight (<30 kg) and was deployed from small vessels by a team of 2-3 people depending on operating conditions. For example, in conditions that allowed the vessel to be anchored or berthed at pontoons/wharfs, a two-person team was sufficient for safe operations, but during un-secured operations a three-person team was required to safely operate the vessel, the ROV, and manage the ROV’s tether.

The SCUBA diving personnel used for this study have experience in marine biosecurity surveillance across ports and harbours throughout New Zealand (Woods et al., 2018). These specialist scientific diving teams were regularly trained on the identification of high-risk marine pests, were familiar with many of the NIS present in New Zealand and maintain a high level of knowledge of native species present throughout New Zealand. Divers used in this study had a minimum of five years’ experience. Furthermore, more experienced divers (> 10 years’ experience) were always paired with less experienced divers. Although the same divers were not used for every dive in every region (on account of dive profile management and regional diver availability), the experienced diver was always denoted “Diver 1” while the less experienced diver was denoted “Diver 2”. Like ROV surveys, dive teams used artificial lighting (e.g., high powered LED torches) to illuminate their surroundings and aid in the identification of organisms.

Divers had a primary objective of detecting nine target marine pest species, five that have yet to be detected in New Zealand (the Northern Pacific seastar Asterias amurensis, the European shore crab Carcinus maenas, the green seaweed Caulerpa taxifolia, the Chinese mitten crab Eriocheir sinensis, and and the Asian clam Potamocorbula amurensis), and four established pest species (the Asian date mussel Arcuatula senhousia, the droplet tunicate Eudistoma elongatum, the Mediterranean fanworm Sabella spallanzanii, and the clubbed tunicate Styela clava). However, as a secondary objective, divers are tasked with detecting non-target NIS know to be in New Zealand waters (e.g., the Asian paddle crab Charybdis (Charybdis) japonica, the colonial ascidian Didemnum vexillum, and the kelp Undaria pinnatifida) and any suspect organisms thought to be new to New Zealand (Woods et al., 2018). A selection of these species, and other NIS commonly found in New Zealand are shown in Figure 2 . When comparing the species detected by each method, we separately analysed the detection profiles for total NIS observed (i.e., primary and secondary objectives) and the target subset of organisms (i.e., the primary objective of nine target species and the three non-target species).

A key aspect of the survey protocol that these programmes have implemented for 20 years was the focus on detecting new incursions of NIS, not enumerating densities or abundance of established NIS (Woods et al., 2018). This strategy was implemented to ensure that human observers did not become overwhelmed counting abundant NIS but maintain mental capacity to observe and detect primary or secondary target species. Therefore, we assessed the efficacy and efficiency of divers and ROVs at detecting the presence or absence of NIS and do not report densities or abundances. However, we briefly report on whether NIS were identified by single specimens or abundant populations.

Kaipara Harbour ( Figure 1 ) presented a strong turbidity gradient, from the upper arms of the harbour (turbid) to the harbour entrance (clear). We used this gradient to compare the swath of benthos sampled by divers and the ROV at high visibility (> 3 m secchi), moderate visibility (1.5 m secchi) and low visibility (0.8 m secchi) ( Figures 1A–C ). Two sites were investigated at each turbidity range. At each site all NIS were noted, and the time and distance covered recorded. The width of seafloor within clear view of the ROV and SCUBA divers was estimated using parallel lasers to calculate the average frame width (ROV), and half the secchi disk measurement taken at the surface (divers). While it is not appropriate to assume that the benthos could be accurately sampled by divers at the full length of the secchi measurement, we considered that half the secchi depth was a conservative swath width of human divers. Wharf piles were surveyed at locations of high and low turbidity (no wharf piles were found in regions of moderate turbidity), and the times taken to cover these features were also recorded. All NIS detected were enumerated, and any suspected NIS or unknown native species were sampled by SCUBA divers. Samples were preserved according to the taxon to which they belonged (as identified by trained parataxonomists) and sent to specialist taxonomists for formal identification.

The average point-to-point distance covered per unit time, by the ROV and divers, was estimated from the difference between the start and end Global Positioning System (GPS) coordinates and the time it took to cover the distance. The GPS coordinates were collected with a Lowrance™ HDS-16 chart-plotter with horizontal accuracy < 5 m.

ROV and SCUBA surveys were completed at all three locations ( Figure 1 ). The 14 sites (Kaipara, four sites; Nelson, six sites; and Whakaraupō/Lyttelton, four sites) surveyed included a range of natural and artificial habitats including rocky riprap/reef, soft sediments, man-made pontoons, and wharf piles. Sites were pre-determined and were assessed by ROV first to avoid missing specimens collected by divers. SCUBA surveys were completed no more than two weeks after ROV surveys. Sampling was completed during daylight hours at each location. Sampling in Kaipara Harbour was completed during the week 20-24 May 2019; between 18-26 June 2019 in Nelson; and between 1-15 July 2019 in Whakaraupō/Lyttelton Harbour.

At each site, a defined area was sampled in line with the NMHRSS programme (Seaward et al., 2015), either 50 m benthic transect, 50 m pontoon structures, or ten wharf piles were sampled, and the time taken for each method recorded. Divers searched the prescribed area (e.g., linear distance travelled or prescribed number of wharf piles) in “buddy” pairs, swimming the same route which included observations of both overlapping and discrete habitats. Furthermore, all NIS observed by divers or ROV were recorded. The sensitivity of each method (i.e., the proportion of NIS, sampled by each diver and the ROV, compared to the total NIS sampled by all methods) and the detection rate (i.e., number of NIS sampled per minute) were calculated for each site.

Sensitivity (Equation 1) was calculated as the total number of NIS (number of species, not number of individuals) found at that site by each method (M_s ) compared to the total number of NIS observed by all methods (i.e., both divers and ROV) at that site (S).

Equation 1:

Detection rates (Equation 2) were calculated as the total number of NIS detected by each method (M_s) divided by the time taken to complete the survey (T_min). Detection rate was expressed as NIS detected per minute.

Equation 2:

Observations of NIS were recorded in situ by divers. However, ROV videos were processed following completion of field campaigns by trained parataxonomists with no prior knowledge of the observations made by the divers. In cases where NIS were suspected, but were unable to be confirmed via specimens, confirmation by diver collected specimens was used to confirm presence. We note that without performing a full biodiversity census at each site we were unable to assess the inability of both methods to detect NIS present within a site.

Species assemblages at each location and for each method (including separate detections from each diver) were used to identify differences in taxonomic profiles detected by each method. The influence of substrata, method, and region on NIS composition was examined. We analysed the total richness of NIS detected by both methods as well as the subset of primary target species to ensure that additional capability to review ROV video did not bias the detection of additional species compared to divers.

We assessed the relative efficiencies and efficacy of diver and ROV methods, including the costs of manual footage review compared to artificial intelligence (AI) augmented processing of ROV video. This was done by summing the time required to cover 1 hectare (ha) of seafloor and multiplying this by the number of people required for operations. Four people were required to complete diving operations, while three people were required for ROV operations. ROV surveys required post-processing of video to analyse the presence of NIS and this time was added to the total time costs of ROV surveys. Video processing time was added to the survey cost at the same ratio as the footage collected (i.e., one hour of benthic video = one hour post-processing). We note that equipment (e.g., ROV, dive gear, vessels) and consumables (e.g., fuel) was not factored into calculations of costs. We focus on the personnel costs of completing surveys after investing in the appropriate equipment.

To demonstrate the applicability of AI detectors we trained and tested a computer vision algorithm against expert marine biosecurity specialists. Here, we examined the potential for computer vision algorithms to recognise and discriminate a key unwanted, yet widely present invasive species of tubeworm, Sabella spallanzanii, including testing against a similar native tubeworm Pseudobranchiomma grandis (both Family Sabellidae; Figure 3 ). S. spallanzanii has been shown to be damaging to ecosystem services and marine industries (Soliman and Inglis, 2018; Atalah et al., 2019; Douglas et al., 2020; Tait et al., 2020). Halting regional spread of species such as S. spallanzanii is a key goal of regional and central government agencies across New Zealand and Australia (Cunningham et al., 2019; McDonald et al., 2020).

We explored the potential for automatic NIS detection in survey videos (collected with the Boxfish™ ROV), using AI to identify a NIS already present in New Zealand. We trained a YOLOv3 AI model to detect the non-indigenous tubeworm, S. spallanzanii and the indigenous tubeworm P. grandis. This classifier was chosen because of its suitability for object detection, and in particular its utility for real-time detection.

YOLOv3 (You Only Look Once version 3), is a real-time object detection algorithm that identifies specific targeted objects in videos (as a post-process). During training, YOLOv3 iteratively learns the features required to accurately identify its target and, after each iteration, discards any information that does not improve the accuracy. To train the YOLOv3 model we used a small dataset of approximately 100 individual images of non-indigenous tubeworm S. spallanzanii. For testing, we used independent ROV video that had not been used for training.

From our selected ROV survey video, we fed a set of 100 frames (set A) into the trained detector. When the detector predicted the presence of the target (S. spallanzanii) anywhere within the frame, it drew a coloured box around (annotated) each instance of the detected target and saved the set of 100 annotated frames (set B) for later analysis. Set A was given to one of two experts to manually count and record the number of individual S. spallanzanii that were visible in each frame. Set B was given to the second expert to count and record any AI-annotated detections in each frame. We then carried out a comparison between the instances of S. spallanzanii manually counted in set A and those automatically detected in set B. While the trained detector algorithm included both indigenous and non-indigenous tubeworms, only the invasive S. spallanzanii was present in the test video.

Variation in sensitivity and detection rates between methods (including between divers) were analysed using one-way ANOVA, including post-hoc Tukey tests. Diagnostic plots were used to check for outliers and homoscedasticity before Tukey tests were performed. Statistical tests were done in R Studio (RStudio Team, 2020).

Composition of NIS was analysed using principal coordinate analysis (PCA) and permutational ANOVA (PERMANOVA) using the R ‘vegan’ package (Oksanen, 2007). We chose multivariate analysis to retain species specific information and identify subtle biases in the detection profiles of each method. To test for and visualise these responses we overlayed the contribution of individual species using redundancy analyses to the total species profile of each replicate (i.e., a single site by one method). The Jaccard index was used for calculating dissimilarity indices for the presence-absence data. The combined effects of method (Diver 1, Diver 2, or ROV) survey substrata (rocky reef/riprap, soft sediment, or pontoons), and location (Kaipara Harbour, Nelson Harbour, and Lyttelton Harbour/Whakaraupō) on the composition of NIS detections were analysed using PCA and PERMANOVA. PCA plots included vectors representing the contribution of NIS to detection profiles and frames surrounding treatment groups.

Agreement between computer vision detection algorithms and expert observations were analysed with linear regressions. To test the deviation of the regression slope from one (i.e., perfect agreement) the difference between computer vision detections and experts’ detections were compared to expert detections. Significant deviation of the regression slope from zero was used to assess the over- or under-detection of S. spallanzanii by the computer detector.

Under decreasing water clarity, the area of seafloor or surface area of structures surveyed per unit time by divers or ROV decreased dramatically ( Table 1 ). This was determined by the area of seafloor/structure visible under a gradient of turbidity, where decreasing water clarity affected the proximity of the ROV to the benthos/structure and therefore the area of benthos clearly visible per linear metre of benthos/structure covered or the number of piles covered per unit time. Since it was necessary for SCUBA divers to dive as buddy pairs for safety reasons, the area of benthos/structure sampled was doubled, but the area surveyed by a single diver and the ROV was comparable.

Combined, visual surveys with ROVs and divers detected 35% of all known NIS present in Kaipara Harbour, 14% of all NIS known from Nelson Harbour, and 15% of all known NIS from Whakaraupō/Lyttelton Harbour. ROV alone detected 28% of all known NIS present in Kaipara Harbour, 8% of all NIS known from Nelson Harbour, and 10% of all known NIS from Whakaraupō/Lyttelton Harbour. The sensitivity of detection was similar for each method, with no statistical difference in the proportion of the total NIS detected by each method ( Figure 4A ). However, detection rates (e.g., the number of unique NIS sampled per unit time) of NIS were significantly greater for divers compared to the ROV ( Figure 4B ; Table 2 ). Tukey post-hoc tests revealed no significant differences in detection rates between Diver 1 and Diver 2, or between Diver 2 and ROV, but Diver 1 had higher detection rates than ROV ( Table 2 ).

The composition of NIS assemblages as detected by each method and at each site were presented in two-dimensional space as Principal Coordinate Analyses (PCA). To visualise the overlap in detection profiles we present duplicate plots highlighting the differences between each method ( Figures 5A, B ), various substrates ( Figures 5C, D ), and regions ( Figures 5E, F ). Principal coordinate plots showed high overlap in the detection profiles between all methods for the full suite of NIS detected ( Figure 5A ) and the subset of targeted high-risk species ( Figure 5B ), although ROV detections captured only a subset of the overall NIS profile observed by divers ( Figure 5A ). Unlike survey methods, NIS profiles were distinct across substrate types ( Figures 5C, D ), and across regions ( Figures 5E, F ). PERMANOVA analysis showed that the species profiles detected by each method were not significantly different, but the species profiles differed significantly between substrata ( Table 3 ).

Species poorly detected by the ROV were relatively small organisms, including the skeleton shrimp Caprella mutica and the hydroid, Ectopleura crocea ( Table S1 ). Furthermore, the ROV did not identify any of the Asian paddle crab (Charybdis (Charybdis) japonica) in Kaipara harbour which were observed by divers at the same location in sampling 24 hours apart ( Table S1 ). However, the invasive goby Acentrogobius pflaumii was only detected by the ROV and not by divers at the same location one week later ( Table S1 ). It is worth noting that while mobile organisms such as the crab Charybdis (japonica) japonica may have simply moved over 24 hours, the tendency of Acentrogobius pflaumii to hide in burrows means it may simply go un-noticed if disturbed.

The relative costs of diving and ROV assessments in terms of the number of people hours required to cover one hectare (ha) of seafloor showed that ROV methods were 1.8 times more expensive than divers and 2.4 times more expensive than divers if manual imagery review was required ( Table 4 ). Both ROV and diver searches in low visibility environments (0.8 m secchi) were almost four times more expensive than searches in clear water (> 3 m secchi). The level of visibility had little impact on the relative expenses of ROV and diver-based methods ( Table 4 ).

Computer vision algorithms had no misidentification of non-indigenous tube worms as indigenous and vice versa. The independent test ROV survey video had high numbers of the non-indigenous tubeworms at various distances from the camera, providing a relatively challenging task for automated algorithms and experienced observers. Computer vision algorithms showed high agreement with expert-based video observations for the enumeration of the non-indigenous S. spallanzanii ( Figure 6 ). There was, however, a trend of false negatives by the automated detector. Linear regression of the differential between detection by experts and the detector and the total number of specimens observed by experts showed a significant negative trend (t = -9.7, p< 0.001), indicating that increasing numbers of specimens lead to increasing ratios of false negatives by the detector. Higher agreement at lower densities shows that automated detectors provide fewer instances of false positives when presented with low numbers of Sabella spallanzanii. Overall, there were only three instances where computer vision detections exceeded expert detections by a single specimen.